Questions tagged [distributed-computing]
The distributed-computing tag has no summary.
43 questions
0 votes
0 answers
72 views
Zipping an S3 folder across AWS Batch containers
I am building a data pipeline where the final step for each submitted job will be zipping all the files inside the job folder, sitting in an Amazon S3 bucket, into a single zip file. I'm evaluating ...
- 101
3 votes
1 answer
285 views
sge: How to set hard limits of node?
The guy who administrates our cluster died suddenly and recently, so now we have to operate it ourselves until a new guy comes in. We want to change the hard runtime limit of a node on our cluster. ...
- 85
-3 votes
2 answers
189 views
Heterogeneous Cluster Solution for R&D
I work in a research laboratory with multiple physical machines with different specifications. The machines have different CPUs (some Intel, some AMD), different RAM sizes, some have discrete GPUs, ...
- 55
0 votes
1 answer
112 views
Good technology for a large-scale batch operation on many S3 files in AWS Batch with Spot instances
I have an enormous corpus of text data stored in millions of files on S3. It's very common that I want to perform some operation on every one of those files, which uses only that file and creates a ...
- 99
0 votes
0 answers
31 views
What are my options for a distributed, row-oriented database?
(Sorry if this isn't the right forum. StackOverflow doesn't seem right, either. I thought there used to be a description on the home page saying what kind of questions were relevant but I can't find ...
- 119
2 votes
1 answer
2k views
Can kubernetes cluster nodes be geologically spread out?
Me and a couple of friends are thinking of setting up a kubernetes cluster where our homeservers will act as the nodes. As our nodes will be spread out between our apartments I am worried this will ...
-1 votes
1 answer
203 views
Volunteer Distributed computing like Folding@home
I'm a user of Folding@home and like its ability to leverage idle machine to process tasks. I'd like to do something similar with my work where I can allow users to volunteer their machines and have a ...
- 103
1 vote
0 answers
257 views
Is it practical to keep home directories on OpenAFS?
I am a sysadmin by trade, and I do what I do at work at home as well for fun. I have a Gentoo Linux laptop, Raspberry Pis running Raspian, a Gentoo server, ARM devices running Debian and have various ...
- 1,008
0 votes
0 answers
23 views
Need to convert 1000 images in 5 seconds-- what are our options? [duplicate]
we're looking for a server architecture that will allow for converting 1000 large images in 5 seconds. As a test, we ran some benchmarks using a 16 core server, using GNU Parallel to run 1,000 image ...
- 111
0 votes
1 answer
3k views
How to manage failover in ZooKeeper across datacenters using observers
I have an application running in 3 different datacenters that uses ZooKeeper for many tasks. Following the recommended practices, we have deployed three ZooKeeper ensembles where one datacenter ...
- 191
-3 votes
1 answer
275 views
Solutions for monetizing excess CPU cycles [closed]
My company has a big (relatively) computer farm, say, 100 physical servers (dual CPU hexacore e5 xeons with 160 Gb RAM) leased from some hardware provider (say Leaseweb or OVM) on monthly basis, means,...
- 195
4 votes
2 answers
2k views
How do I distribute HTTP traffic to the closest server?
This is kind of a two part question. So I have a DigitalOcean Droplet in Toronto with a lamp stack on it (with a website of course). I want to be create a snapshot of that droplet and deploy a clone ...
0 votes
1 answer
4k views
Nginx as GeoIP Global LoadBalancer and actual distance from app server
my client is using Magento for it's ecommerce global solution. Main country are US / EU / CN. For now only 1 VM in Italy, serving all traffic. I would like to create 3 VMs, one for each country, on ...
- 302
2 votes
1 answer
109 views
distributed storage solutions [closed]
I've got some desktop PCs available in my company since the last user hardware renewal and I would like to re-use them for a demo plaform. They are powerfull enought to emulate a few servers. I ...
- 131
1 vote
1 answer
139 views
Distributed image processing on AWS: how to know the address of master node?
Context: I am building a mobile application that will let users apply advanced filters to their pictures. Since the processing can't be done on the phone, it will be offloaded to EC2 instances. I ...
- 313
1 vote
2 answers
124 views
Server utilization and how to deal with datacenter failure
New technologies like Docker, Mesos, Kubernetes etc. allow much better server utilisation in an organisation. However, I'd like to know how utilisation can be maximised across two datacenters, taking ...
- 135
16 votes
5 answers
11k views
Are there any Distributed/mesh-like/P2P VPNs?
I have my personal VPN, connecting several devices so they can have fixed IP addresses on an always-reachable network (as long as they are connected to the Internet). This is necessary for me, since ...
-2 votes
3 answers
2k views
Is there any standard for DNS <-> IP authentication?
I'm looking to implement an authentication mechanism which allows to enforce access policies based on the domain name of the client. The authentication server uses the information available in the DNS ...
- 335
1 vote
2 answers
269 views
Distributed storage and computing [closed]
Dear Serverfault community, After researching a number of distributed file systems for deployment in a production environment with the main purpose of performing both batch and real-time distributed ...
-1 votes
1 answer
58 views
Is there a way I can share system resources across a heterogenous system? [closed]
I have two machines, one is a laptop running windows 7, the other a desktop running fedora. Is it possible to connect the two computers together somehow so that the laptop uses the memory or disk ...
- 7
1 vote
2 answers
2k views
When should an EC2 instance use "Amazon instance store-backed"?
What are the benefits over using Amazon instance store-backed over EBS block storage? The only thing I can see from the comparison of the two is that instance store-backed seems to incur less charges. ...
- 1,311
-3 votes
1 answer
326 views
What to do unused servers [closed]
Since I have an application that is growing and i run on hardware I always have a few extra servers sitting around and want to find away to donate them. I am famulaure with folding@Home but maybe ...
- 350
4 votes
1 answer
3k views
How to set up alerts on Ganglia?
How can I set up Ganglia so that I get an email if a machine in the cluster is using, for example, greater than 95% of physical RAM?
- 143
0 votes
1 answer
63 views
How to distribute multiple executions of an app across many machines
I've got a simulation app (64-bit windows) that runs without any user interaction. This app gathers information and pushes it to a remote MS SQL Server. What I'd like to do is execute this ...
- 101
1 vote
3 answers
1k views
Distributed computing through AWS
I know it's possible to balance traffic using ELB/EC2 instances but is there any way to distribute CPU calculations over a cluster of instances. I'm wanting to process lots of video/audio using ...
- 382
2 votes
5 answers
2k views
Distributed cron
I maintain many servers and have the following requirements Run a unix process at a given time Run a unix process at boot time and keep it running (in case it dies) Disable a running process I am ...
- 145
-3 votes
2 answers
1k views
Parallel computing: Distributed systems vs multicore processors? [closed]
I was just wondering why there is a need to go through all the trouble of creating distributed systems for massive parallel processing when, we could just create individual machines that support ...
5 votes
1 answer
1k views
Distributed and/or Parallel SSIS processing
Background: Our company hosts SaaS DSS applications, where clients provide us data Daily and/or Weekly, which we process & merge into their existing database. During business hours, load in the ...
- 51
0 votes
2 answers
155 views
Distribute the load all around the world
Let's say we're starting a project which is meant to be served all around the world How am I supposed to distribute the database and server load, and increase (optimize) the service for large ...
- 271
0 votes
2 answers
2k views
Calculating the power of a cluster vs a single workstation [closed]
For the same price we can get either Worktsation which is 2 x 6 core Xeons 3.4GHz, 128GB RAM, 600GB SAS RAID 0 = 12 Cores, 128GB RAM Or Cluster 8 x (4 Core i7 3.4GHz, 16GB RAM, 250GB SATA, Intel ...
- 139
0 votes
1 answer
315 views
Using Opennebula as an IaaS: Running benchmark accross all nodes, possibe?
Our current situation: We have one Opennebula cloud infrastructure consisting of a cluster controller and 6 cluster nodes configured and working. All Nodes run debian instances on top of a kvm ...
- 13
0 votes
2 answers
172 views
Condor Compute Cluster: quick information about Distributed Systems? [closed]
If you guys have never heard of the idea behind Condor Cluster: http://www.cs.wisc.edu/condor/ It's developed by the University of Wisconsin but used by many other universities since its birth. I was ...
- 97
0 votes
2 answers
257 views
Good books on engineering distributed systems
I am looking for a good book / blog / resources on Engineering Best Practices (developing, testing, automation, tools & deployment) for distributed systems. Most of what I see is principle, ...
- 11
3 votes
2 answers
837 views
What is an easy solution to distribute commands/tasks in a Linux network? [duplicate]
Possible Duplicate: Linux - Running The Same Command on Many Machines at Once Here's the situation: We have a lab that consists of fifteen quad-core machines, each running Ubuntu Linux. There is a ...
- 341
6 votes
6 answers
1k views
Huge File System?
Suppose that one has 100 machines (each with a 2TB hard drive), all connected in a network. Is there a way (in Linux) to unite this combined 200TB of disk space into one folder, that can be shared ...
- 231
4 votes
1 answer
768 views
Recommendations for distributed processing/distributed storage systems
At my organization we have a processing and storage system spread across two dozen linux machines that handles over a petabyte of data. The system right now is very ad-hoc; processing automation and ...
- 363
1 vote
2 answers
269 views
Build parallel processing grid from 32-bit desktops
I am running simulation on my laptop(Core 2 Duo 1.8Ghz with 4GB RAM running windows 7) which is taking very long time. I have a couple of unused 32-bit (Core 2 Duo 1.8Ghz with 4GB RAM) desktops. I ...
- 113
0 votes
2 answers
837 views
How to use combined CPU/Memory power of a Windows cluster
I have 5 Windows machines (dual-core, 3GB) in a LAN all joined to a domain. I have a program which needs 8-cores and 10 GB to run in a given SLA time. What platform/tool can i use to harness the ...
- 111
0 votes
1 answer
94 views
Per-thread Distrubuted Computing? [duplicate]
I want to run an unmodified MS Windows application, distributed across multiple machines. Can I extend its Symmetric Multiprocessing support to span multiple CPUs on multiple machines? Instead of just ...
- 1,972
3 votes
4 answers
288 views
University Renderfarm management tools?
Faculty are requesting tools to manage the rendering crunch during finals week. Currently the status quo is a free-for-all in the computer lab during project deadlines. People end up "reserving" ...
- 14.6k
6 votes
3 answers
574 views
Load-balanced Linux server across internet?
I'm investigating setting up a load balanced server solution consisting of three CentOS 5.4 boxes. Two of these boxes will reside in one facility, while a third will reside in a different facility. ...
- 237
2 votes
4 answers
216 views
Distributed computing for a company? Is there such a 'free' thing?
I am new to the whole distributed computing / cloud thing. But I had an idea at work for our multimedia stuff like movie encoding / cpu intensive things tasks (which sometimes take a few hours). Is ...
- 380
5 votes
4 answers
3k views
Is there something (free) like ZFS or HadoopFS or GFS for Windows?
At work, we've got a bunch of boxes with free HD space. I'd like to run something like ZFS on those machines, making a large virtual file system available to all of the users. In my mind, it would ...
Matt Cruikshank