Four instances EC2 't2.medium' all upfront (3 year paid in advance) = 4 x $607 = $2416 (~€2200). A nice target for a Bitcoin Campaign
Load balancer = ~$19 (~€18) per month

So the database (db.r3.xlarge, all upfront for 3 year = $5314 (~€4863) is the major part of all costs? That is a very big target, lot's of ASICS needed...

As Willy predicted the aws-costs are higher than the metal-costs.
But yes, the further testing by Willy is more important now.

The monthly costs are higher, but AWS should have no maintenance costs.

Last year I had to replace four broken hard drives and a broken RAID controller. If BS doesn't move to AWS I have to replace all the SSD's.

And I can predict that the metal servers will not live forever. They are four years old now. Murphy says they will brake down eventually.

Daily update ran without problems, albeit with outdated data.

From today on and if nothings goes wrong the host update will run in production on AWS. Unfortunately, you can't see the results here.

Next up is running the daily user/team/country update in production and if that goes well too and nothing dies and Amazon doesn't ask me te leave (* there's a story here) then I'll start working on moving the remaining updates (team and project challenges) to AWS and the web part of BOINCstats.

Story??

BOINCstats actually started on shared hosting, some sort of predecessor of cloud. Back then there were two projects (SETI and Predictor) and I had the first version of Combined stats. No host stats yet. Compared to now there were 1/10000th of the calculations we do now.

After about two weeks I was kindly asked to leave because the host servers could not cope with the load. I bought the first metal after that.

And now BOINCstats may return to completely virtual again.

User, team, country and host updates are now successfully running. Going to optimize things now, see if performance is acceptable. As it is now the updates run for too long.

Cost may however be an issue. This month so far (remember, at the time of writing this the month is not even a day old) has cost $32.11. My calculations (pretty simple, multiply by 31) make that a total of $963.30 per month, totally unacceptable. AWS however predicts a monthly total of $256.38. That's reasonable but I don't get how they get that number. Cost may be a bit higher now than in production because of all the tests I do, data imports of multiple GB and stupid mistakes I made.

Without mistakes and errors: No Learning

I'd be worried about the runaway data costs when/if web servers go live and start getting hit by the thousands, serving pages, charts, forums, and BAM! connections...

I spoke with the DC people and they agree to let me go with a two months notice. So I can keep the old configuration standby even when we move production to AWS. If it should really get out of hand then we can move back. If it succeeds it's "only" €500 euros wasted (for who's wondering, that's the bare price for two months without paying for energy).

Also, I think that the stats updates generate more queries on the server than the users visiting the page. There are almost four million users in the stats database, over 16 million hosts (both duplicated because of the no-asic stats) while only a couple of thousand users visit BOINCstats each day.

Some server stats from the AWS database server:

This MySQL server has been running for 5 days, 10 hours, 18 minutes and 42 seconds. It started up on Dec 28, 2015 at 08:34 AM.

Traffic 	#	        ø per hour

Received	520.7 GiB	4 GiB

Sent	        413.5 GiB	3.2 GiB

Total	        0.9 TiB	        7.2 GiB

For comparison, the current update server:

Traffic         #	        ø per hour

Received	7,237 GiB	13 GiB

Sent	        9 TiB	        17 GiB

Total	        16 TiB	        30 GiB

Two days in this month, cost so far $94.83. 94.83/2*31 = $1469.

This is the culprit:
$0.20 per 1 million I/O requests (Aurora) -> 214,982,108 requests -> $43.00
$0.58 per RDS High Memory R3 Extra Large Instance hour (or partial hour) running Amazon Aurora -> 43 Hrs -> $24.94

The second one is the one that can be paid upfront. The first one should decline once I stop messing things up.

But, if these rates hold AWS is not feasible.

Hang in there and play it through, until there is a settled Site there are no definite's. As you pointed out before, until two months stats flow, its hard to be definitive.... albeit suspicious

.... if they did give you a "get out of jail free card" after two months, there is an escape clause.

Don't want to create an uproar here, but how much would it save to eliminate the "without Asics" stats? Although (never having run BU) they are the only ones I ever use, I for one could live without them if the savings is significant enough.

I have no love for the no-asic stats either but I can understand why people want them. It's also not the biggest problem. Host-history is what needs the most time during an update, several hours.

Going to pull the plug on AWS. It's now at $159.98. Host update takes 20 hours to complete, daily update takes 2 hours longer than on metal. And that's will several attempts to optimize everything. This won't work, at least not at an acceptable price.

Shutting everything down now, letting it sink in a bit and then later delete the whole thing.

The good news is that some optimizations can be used on the metal.

Oh, well. At least now it has been tried...

Thanks for all the diligent work!

1 and ! Web Hosting .... Worth a chat with these guys if you are still in the game for Data-Center Sites. They are World No 1 Provider now, mainly based in Europe on your doorstep, and started up a US subsidiary a few years ago - not that it matters where its hosted with 1and1 as they back up Sites to different Continents, their Internal Data Links are massive.

They have hugely expanded compared to a few years ago, especially in the Business Sector.

Worth a chat with their Sales Guys .... they have internal load balancing which you have access to thats going to help with your unequal "box needs"

Cloud servers are free for a month to trial needs et al.

1 and 1 Web Hosting

Regards
Zy

+1

I looked at 1 and 1 Web Hosting and let's configure what I had at Amazon (as close as possible):
2x Cloud Server XL = £40 (update servers)
1x Cloud Server 3XL = £130 (database)
-------------------------------------
£170 + 20%VAT = £204 -> €300 / Month

This is not enough, webserver is missing. They will be another £20.
===============================================================================================
Going at it another way, with dedicated servers:
Server o12A-32 (AMD 12 core, 32GB, 2TB SATA RAID 1 and 240GB SSD RAID 1: £140 (database + update server)
£140 + 20%VAT = £168 -> €247 / Month

Server L4 (AMD 4 core, 4GB, 750GB SATA RAID 1 (webserver): £40
£40 + 20%VAT = £48 -> €71 / Month

Total €318/month which is about the same as the current setup.

This requires the same changes to the update code as I did for Amazon since there will only be one database.
This also has the same disadvantage as the current setup: if something brakes down in the dedicated server it needs te be replaced and depending on what broke down there could be data corruption. And downtime during repairs. So the biggest advantage is that I don't have to do the repairs to the hardware.

Thats the big advantage of 1and1, there are no physical hardware costs for you to Own. The latter is baked into the monthly price - its actually run on massive UNIX Racks and Mainframes et al.

All the backups are done automatically by them, and recovery etc - its a fully managed service. There are no other costs other than increase in storage and data traffic. There is no down time (in theory) as its all automatically recovered/mirrored/managed for you - the latter is very attractive on its own.

You dump all hardware costs - forever - and have no worries over system recovery other than at the application software level, they do all system recovery if it crashes (which it will not at a System level - its on their fully mirrored Systems - they dont want the cost of recovery ....). Hardware worries are taken out of the equation completely. Operating System Level is therefore taken care of for you, all you need to do is focus at the Application / User Level because all else is done for you.

Bound to be some disadvantages in the detail of application interface with the Operating System as you did your own, but I cant see that being a show stopper as other advantages on Managed Service outweigh that slight disadvantage. At the end of the day all network, Operating System, recovery, and power/fuel generator costs are baked into their offering. They are very flexible on System Design, there is a "Pick and Mix" system at the bottom of the page I referenced.

You may need to chat with them about whether or not the Dedicated Servers are needed, frankly I would not have thought so, I would have thought their Virtual Servers would eat this with ease - accepting some limitations on configuration, but I cant see the latter being a Show Stopper. The UK Site is not that far from where I live, the building is Massive, multi-story, huge data storage and multi-generator backups on an Industrial Scale, all air-conditioned, and security like Fort Knox ..... a very, very impressive setup. The Mainland European Sites are just as big. All interconnected by massive Data Links - the latter is how they keep the flexibility and costs down, they use other Countrys 1and1 Sites to store backups et al and provide Auto-Failover.

From that initial look at the costings, I would say it is worth pursuing with 1and1 Sales ..... see how it goes, if it looks a Go after the initial chat, you still have the "get out of jail free card" of pulling out after one month at no cost. Apart from investment of time and hassles, you will lose nothing in that month.

Response time is an unknown as such, but from what I have experienced with them, for the money its very good considering its a Shared Managed Service....... and bare in mind this Service is on offer to SME's so it has to be good.

You lose nothing by chatting with them.

Regards
Zy

I've send 1and1 an email explaining the situation and needs.

I guess another thing to bare in mind - and its "the elephant in the room" - is what the future holds for BOINC. The advent of Bitcoin Utopia has obviously had a massive impact on BOINC, and thats not going away anytime soon.

The 1and1 Managed Service also holds - I would have thought - other attractions for you in this regard, as you can scale down Server, Processor, Memory and Storage Space costs, if BOINC gets any quieter (equally ramp it up if it does increase in activity) and not be caught with expensive hardware investment that gets overtaken by fluctuation/decrease in User Numbers. You are therefore not in danger of having redundant hardware in the future that cost you hard cash to setup and/or upgrade.

You would be able to "go with the flow" as they say, and not get caught out by extensive increases/decreases in User numbers as the Bitcoin Utopia story plays out over the next few years...... but still have the ability to stay flexible to provide for BOINCStats needs by up-scaling/down-scaling the Virtual Service used at 1and1, therefore keep unexpected costs under control....... and equally, easily scale down hard-ware and costs a little, if activity continues to decline.
Regards
Zy

Hi,

to me it sounds that you want to make a lot of promotion for 1&1...? I have only little experience with 1&1, but their servers are quite expensive. To the points you mentioned: All hosters offer more or less features that you have to pay for. There is nothing special about 1&1 imho.

Funny thing is that the servers (which Willy looked at) have different prices for me in the UK store:

Cloud Server XL = £30 (you price was £20)
Server o12A-32 (AMD 12 core, 32GB, 2TB SATA RAID 1 and 240GB SSD RAID 1: £130 (your's was £140) - as far as you did not chose the "Managed" option

All prices excl. VAT.



The german prices are:

Cloud Server XL = 30€
Cloud Server 3XL = 130€
Server o12A-32 (incl. 240 GB SSD RAID) = 150€ (+ 10€ for "Managed&quot
Server L4 = 50€

All these prices already include %19 VAT! Nevertheless it's still a little expensive for my taste.

But there are sooo many hosters on the market... it takes time (and money ) to find a suitable "solution".

I dont give a rats what Provider ....

Its the principle behind it thats more important. The provision of a Managed Service is far more in line with BOINCStats needs over the medium term than Owned self-financed Servers.

1and1 are merely a good example of what can be done with Managed Services. I strongly suspect Willi is capable of working out which Provider is best to use, if any.

Willi will make up his own mind on what he finds, no matter what we say, he's the one in the Hot Seat