I'm trying to get BOINCstats working on Amazons' cloud platform. It's pretty expensive so I'm starting all tests with the cheapest virtual machines and/or database instances.

The XML import is working with pretty decent performance. Single project user update also works but only tested a small project since the data isn't transferred yet.

I'm now working on the host update. The database is importing right now and I'm running a query to get the BOINC Combined totals. It's really slow. As this is done purely in SQL I could easily double the database performance, however this also doubles the cost to $431 per month. And that's only the database. The VM to run the stats on also needs to be upgraded, the default only has 1GB of RAM and the dataset for hosts is 4GB.

More later.

Combining host totals. This is without the ranking and history calculations. Just getting the total and average credit for hosts from all projects combined:

Current metal server:
Seconds needed to fetch data: 45
Seconds needed to order data: 396
Seconds needed update table: 756
Total: 1197 second.

On AWS I couldn't run the script because of memory constraints so I reprogrammed it to run completely in SQL:
Seconds needed to update: 2835

Without upgrading AWS this needs ~27 minutes more time to complete.

Everything is ready to do a full host update. However, it's now time to be with my family. And having a little thing to eat. Something like:


Tomorrow morning I'll run the update.

Merry Christmas to Willy & his Family!!! Thank you for all your hard work over the years to keep BoincStats running!

Merry XMas to you & yours!

Tested a host project update on LHC. The update needed a little bit longer than on the metal server but not too much so that's OK.

However, the AWS instance doesn't have enough memory to run the update. I added a 3GB SWAP file and then it ran correctly. Upgrading the AWS instance to have 4GB quadruples the cost for the instance.

SETI host update takes like forever, I think I need to upgrade the instance, will test that tomorrow.

Enjoy your 2nd xmas meal ;-)

Wouldn't that eventually raise the cost from 300 to 1200 per month, if I understood your previous message correctly?

Not really. Different types of servers have different pricing. Databases are the most expensive. The server that needs upgrading means an increase to 40 euros for this instance. I'll probably need four like these plus the database engine.

The quadrupled instance (2 cores, 4GB RAM) is running quite a lot better. The host update is chugging along without the server becoming unresponsive.

I've been testing now for three days, these are the cost so far:
RDS: $29.36
DataTransfer: $1.31
EC2: $0.05
Tax: $6.45
Total: $37.17

RDS = database, EC2 is the server instance (5ct because the initial instance was free for one year).

Not good: combining the hosts for boinc combined took five hours today

What does it take on the metal? Would increasing the cores and/or memory help? Do you know here the bottleneck is from any perf counters they might provide?

About half an hour on metal. In this case it was the database instance which was too slow.

Zzzzooommmmgggg....



Don't know much about (modern) database optimization. What are the options short of data partitioning, incremental caching, more SQL optimizations, more use of precompiled Stored Procedures?

That brought a smile to my face. I've been optimizing this sucker for ten years now.

I already partitioned the queries but no improvement.

Bummer. More cores & memory on the database server? Yeah, would increase the price, but to get an idea...

How is the metal-config (cpu, ram, ssd) compared to the aws-config? Should be more or less similar to get same kind of expected values on the different parts of the update?

The metal server has 8 cores, 32GB RAM and uses MariaDB. The AWS database has 2 cores, 15GB and uses Aurora. According to Amazon, Aurora is five times faster than MySQL/MariaDB. Let's say they exaggerate and that it is three times faster. It could/should be on par with the old server.

Now I expect that it will be somewhat slower. Doesn't bother me that much for the host update. But this is way slower.

The first upgrade to the database server gives it 4 cores, 30.5GB and improved network speed. This doubles the price (pricing).
On demand it goes from $215 to $431 per month. If I could pay three years in advance it goes from $75 to $150 (that's affordable but requires $5314 to be paid at once). Or, yet another option, paying $2500 in advance and $87.60 per month for a total of €160 per month.

Of course, before I pay so much money I want to know if it works but Amazon doesn't see this as testing. If testing lasts a month the full $431 is due.

So far I've spend $55 for testing.

Upgraded the database instance, the host combine update now took 51 minutes.

Now doing the same update for non-asic -> that took 39 minutes.

ok, the aws-test looks like a reasonable performance at a considerable cost: $431 a month is roughly €395, that's more than the current monthly cost for metal?

Well, Willy asks for €300 monthly donation for the maintenance/datacenter costs of the metal, but he said somewhere that they are more like €320-€350 in reality. Plus the Bitcoin Utopia goal of €2100 (excluding the €420-ish campaign late last year), which equals to €175/month. The more realistic total per month for metal would then be ~€500. Plus all the times Willy needs to drive back and forth to replace stuff, buy stuff, hit buttons, etc.

Plus, paying three years up front would seem to bring the AWS cost down to $5314/3/12*0.91=€134/month.

If all these above numbers are correct, then it would totally be worth it. Amazon would take care of all the hardware issues, most times without even Willy hearing about it, he would never have to replace another part himself, and never need to drive to the datacenter.

In any case, it is too early to calculate and compare costs without testing AWS with full data, full functionality and some sort of full user traffic.

Tuna

That is only the database instance. I'm going to need at least two EC2 't2.medium' instances (updaters), two EC2 't2.micro' instances (website, maybe they need to be upgraded) and a load balancer. EC2 pricing, load balancer pricing (BOINCstats transfers about 400GB/month via LB ).

Just ran an incremental update without issues. Speed was fine too. Now trying a daily update.