Questions tagged [distributed-computing]

Question 1

I am building a data pipeline where the final step for each submitted job will be zipping all the files inside the job folder, sitting in an Amazon S3 bucket, into a single zip file. I'm evaluating ...

Question 2

The guy who administrates our cluster died suddenly and recently, so now we have to operate it ourselves until a new guy comes in. We want to change the hard runtime limit of a node on our cluster. ...

Question 3

I work in a research laboratory with multiple physical machines with different specifications. The machines have different CPUs (some Intel, some AMD), different RAM sizes, some have discrete GPUs, ...

Question 4

I have an enormous corpus of text data stored in millions of files on S3. It's very common that I want to perform some operation on every one of those files, which uses only that file and creates a ...

Question 5

(Sorry if this isn't the right forum. StackOverflow doesn't seem right, either. I thought there used to be a description on the home page saying what kind of questions were relevant but I can't find ...

Question 6

Me and a couple of friends are thinking of setting up a kubernetes cluster where our homeservers will act as the nodes. As our nodes will be spread out between our apartments I am worried this will ...

Question 7

I'm a user of Folding@home and like its ability to leverage idle machine to process tasks. I'd like to do something similar with my work where I can allow users to volunteer their machines and have a ...

Question 8

I am a sysadmin by trade, and I do what I do at work at home as well for fun. I have a Gentoo Linux laptop, Raspberry Pis running Raspian, a Gentoo server, ARM devices running Debian and have various ...

Question 9

we're looking for a server architecture that will allow for converting 1000 large images in 5 seconds. As a test, we ran some benchmarks using a 16 core server, using GNU Parallel to run 1,000 image ...

Question 10

I have an application running in 3 different datacenters that uses ZooKeeper for many tasks. Following the recommended practices, we have deployed three ZooKeeper ensembles where one datacenter ...

Question 11

My company has a big (relatively) computer farm, say, 100 physical servers (dual CPU hexacore e5 xeons with 160 Gb RAM) leased from some hardware provider (say Leaseweb or OVM) on monthly basis, means,...

Question 12

This is kind of a two part question. So I have a DigitalOcean Droplet in Toronto with a lamp stack on it (with a website of course). I want to be create a snapshot of that droplet and deploy a clone ...

Question 13

my client is using Magento for it's ecommerce global solution. Main country are US / EU / CN. For now only 1 VM in Italy, serving all traffic. I would like to create 3 VMs, one for each country, on ...

Question 14

I've got some desktop PCs available in my company since the last user hardware renewal and I would like to re-use them for a demo plaform. They are powerfull enought to emulate a few servers. I ...

Question 15

Context: I am building a mobile application that will let users apply advanced filters to their pictures. Since the processing can't be done on the phone, it will be offloaded to EC2 instances. I ...

Question 16

New technologies like Docker, Mesos, Kubernetes etc. allow much better server utilisation in an organisation. However, I'd like to know how utilisation can be maximised across two datacenters, taking ...

Question 17

I have my personal VPN, connecting several devices so they can have fixed IP addresses on an always-reachable network (as long as they are connected to the Internet). This is necessary for me, since ...

Question 18

I'm looking to implement an authentication mechanism which allows to enforce access policies based on the domain name of the client. The authentication server uses the information available in the DNS ...

Question 19

Dear Serverfault community, After researching a number of distributed file systems for deployment in a production environment with the main purpose of performing both batch and real-time distributed ...

Question 20

I have two machines, one is a laptop running windows 7, the other a desktop running fedora. Is it possible to connect the two computers together somehow so that the laptop uses the memory or disk ...

Question 21

What are the benefits over using Amazon instance store-backed over EBS block storage? The only thing I can see from the comparison of the two is that instance store-backed seems to incur less charges. ...

Question 22

Since I have an application that is growing and i run on hardware I always have a few extra servers sitting around and want to find away to donate them. I am famulaure with folding@Home but maybe ...

Question 23

How can I set up Ganglia so that I get an email if a machine in the cluster is using, for example, greater than 95% of physical RAM?

Question 24

I've got a simulation app (64-bit windows) that runs without any user interaction. This app gathers information and pushes it to a remote MS SQL Server. What I'd like to do is execute this ...

Question 25

I know it's possible to balance traffic using ELB/EC2 instances but is there any way to distribute CPU calculations over a cluster of instances. I'm wanting to process lots of video/audio using ...

Question 26

I maintain many servers and have the following requirements Run a unix process at a given time Run a unix process at boot time and keep it running (in case it dies) Disable a running process I am ...

Question 27

I was just wondering why there is a need to go through all the trouble of creating distributed systems for massive parallel processing when, we could just create individual machines that support ...

Question 28

Background: Our company hosts SaaS DSS applications, where clients provide us data Daily and/or Weekly, which we process & merge into their existing database. During business hours, load in the ...

Question 29

Let's say we're starting a project which is meant to be served all around the world How am I supposed to distribute the database and server load, and increase (optimize) the service for large ...

Question 30

For the same price we can get either Worktsation which is 2 x 6 core Xeons 3.4GHz, 128GB RAM, 600GB SAS RAID 0 = 12 Cores, 128GB RAM Or Cluster 8 x (4 Core i7 3.4GHz, 16GB RAM, 250GB SATA, Intel ...

Question 31

Our current situation: We have one Opennebula cloud infrastructure consisting of a cluster controller and 6 cluster nodes configured and working. All Nodes run debian instances on top of a kvm ...

Question 32

If you guys have never heard of the idea behind Condor Cluster: http://www.cs.wisc.edu/condor/ It's developed by the University of Wisconsin but used by many other universities since its birth. I was ...

Question 33

I am looking for a good book / blog / resources on Engineering Best Practices (developing, testing, automation, tools & deployment) for distributed systems. Most of what I see is principle, ...

Question 34

Possible Duplicate: Linux - Running The Same Command on Many Machines at Once Here's the situation: We have a lab that consists of fifteen quad-core machines, each running Ubuntu Linux. There is a ...

Question 35

Suppose that one has 100 machines (each with a 2TB hard drive), all connected in a network. Is there a way (in Linux) to unite this combined 200TB of disk space into one folder, that can be shared ...

Question 36

At my organization we have a processing and storage system spread across two dozen linux machines that handles over a petabyte of data. The system right now is very ad-hoc; processing automation and ...

Question 37

I am running simulation on my laptop(Core 2 Duo 1.8Ghz with 4GB RAM running windows 7) which is taking very long time. I have a couple of unused 32-bit (Core 2 Duo 1.8Ghz with 4GB RAM) desktops. I ...

Question 38

I have 5 Windows machines (dual-core, 3GB) in a LAN all joined to a domain. I have a program which needs 8-cores and 10 GB to run in a given SLA time. What platform/tool can i use to harness the ...

Question 39

I want to run an unmodified MS Windows application, distributed across multiple machines. Can I extend its Symmetric Multiprocessing support to span multiple CPUs on multiple machines? Instead of just ...

Question 40

Faculty are requesting tools to manage the rendering crunch during finals week. Currently the status quo is a free-for-all in the computer lab during project deadlines. People end up "reserving" ...

Question 41

I'm investigating setting up a load balanced server solution consisting of three CentOS 5.4 boxes. Two of these boxes will reside in one facility, while a third will reside in a different facility. ...

Question 42

I am new to the whole distributed computing / cloud thing. But I had an idea at work for our multimedia stuff like movie encoding / cpu intensive things tasks (which sometimes take a few hours). Is ...

Question 43

At work, we've got a bunch of boxes with free HD space. I'd like to run something like ZFS on those machines, making a large virtual file system available to all of the users. In my mind, it would ...