YouTube34m· Nov 2024· cataloged

Building and Running an HPC Grid Using Cloud Native Services - Flamur Gogolli & Kirill Bogdanov


What this covers

This FINOS talk, given by two AWS speakers, walks through HTC Grid, an open-source blueprint for High-Throughput Computing (HTC)) that was donated to FINOS so that customers and other cloud providers can contribute. The first half sets out why financial institutions want to move grids off-premises: market volatility, regulatory pressure, and aging hardware. It places the grid in the middle of a risk pipeline, between data preprocessing and post-processing, and describes the diurnal utilization pattern of on-premises grids, which run full overnight and sit underused by day. From there it turns to the workload itself, which is bimodal. About 40% of tasks finish in under two seconds, and a single institution may submit over 200 million tasks a day. That volume calls for a scheduler handling 30,000+ tasks per second (TPS)), well beyond the 2,000–3,000 typical of conventional schedulers.

The architecture section covers the main design decisions. The principle "all compute is Lambda" means any compute engine or language can sit behind a function-style API. Workers pull tasks from a queue, so the scheduler does not have to check worker liveness. The control plane is fully serverless, at roughly 4,000 lines of code, and is credited with 0.3 seconds of round-trip scheduling latency. The compute plane is modular, with EKS or generic Kubernetes the most common choice. The data plane is picked by workload: ElastiCache for time-sensitive work, S3 for large volumes at low cost, and FSx where POSIX semantics are needed.

The talk also takes some firm positions. It argues that bucketing short and long tasks together is wasteful, and that Slurm tops out at a few hundred tasks per second, so it suits other HPC work rather than this. It advises against automatically spreading work across cloud providers, since no provider guarantees cross-cloud latency. It claims savings from dropping third-party licensing, and it describes isolation by deploying separate copies of the blueprint per team. It closes with planned scaling work: Karpenter, scaling directly on queue depth, and more aggressive scaling algorithms.

Sharpest takeaway

HTC Grid is a cloud-native, open-source blueprint for running high-throughput computing workloads in financial services by using serverless and managed services to achieve 30,000+ tasks per second with minimal operational overhead.

  • Financial institutions face challenges with on-premises HPC grids due to market volatility, regulatory requirements, and aging hardware, making cloud migration attractive
  • HTC Grid uses serverless components and a Lambda API abstraction to handle 200+ million short-duration tasks per day efficiently
  • The modular, cloud-native architecture provides cost savings, operational simplicity, and vendor independence compared to proprietary grid solutions

The claims · ranked22 claims · weighted by value

This asset isn't compiled yet

You're seeing its claims, ranked. Compile it to build the argument threads, weight them, and check each claim against your library — the full view.

0.69

Financial services risk pipelines use compute grids in the middle stage after data preprocessing and before post-processing, with triggers from market closure and trades instantiating different scenarios that need evaluation

factualhigh valueestablishednovelty 1/4durability 3/4· Carol Ivanov

“this is an example of FSI risk pipeline so uh while the actual implementation might be different uh in different fsis but uh the overall concept are quite the same now as the data travels from left to right we have different triggers uh that uh for example the uh Market closure or the trades that uh instantiate different uh events Generations right so different scenarios that needs to be EV Val at now there's in the next stage there is a they merge with the appropriate data that comes from the market from the trades uh all this has been uh prepackaged for the processing and then it's been sent for high performance computer grids”

0.69

On-premises HPC grid utilization shows clear diurnal patterns: overnight workloads (batch jobs) maximize utilization while daytime workloads underutilize the grid, creating customer demand to run more workloads faster without increasing costs

factualhigh valueestablishednovelty 1/4durability 3/4· Carol Ivanov

“if you look at the utilization of high performance comput Grid in the on Prem we can see uh a clear theal patterns so there are different types of the workloads some of them run overnight like overnight B es and some of those workloads run during the day and overnight uh what we usually see is that utilization of the grid is maximized and while at the same time during the day the utilization is uh underutilized right which uh creates um uh the obvious demand from the customers to first of all to be able to run more workloads faster but at the same time don't uh increase the cost”

0.69

Many banks handle the problem of heterogeneous task durations (1 second to 40+ minutes) through 'bucketing': combining short and long tasks to run together so they complete together, but this is inefficient because provisioning compute for a one-second task that then dies is wasteful and doesn't scale

factualhigh valueestablishednovelty 1/4durability 3/4· Dore Goï

“the way most customers chose to deal with this problem is by what's called bucketing which is combining all of these to run together so that they would take longer but ultimately like provision and compute doing a one second thing and then dying is very inefficient it doesn't actually scale right so that's why they kind of wanted to fill that Gap by doing like some some pre pre planning before so they end up running the same slurm”

0.69

Financial institutions face multiple challenges running high-performance computing grids on-premises, including market volatility, regulatory requirements, and aging hardware that make cloud migration attractive

factualhigh valueestablishednovelty 1/4durability 3/4· Carol Ivanov

“when it comes to uh high performance Computing GDs uh there are a number of reasons including Market volatility regulatory requirements and aging Hardware that pose challenges for running the systems on Prem in efficient manner”

0.61

Slurm, a popular open-source HPC scheduler, has task-per-second throughput limitations (reaching only a few hundred TPS, not tens of thousands) and therefore is not suitable for high-throughput financial services workloads despite being appropriate for certain other HPC workload types

factualhigh valueestablishednovelty 1/4durability 3/4· Dore Goï

“has some of those issues because its tasks per second throughput is not as big it reaches a few hundred it can't reach the tens of thousands we're talking about so if you were to use something like slur you can it's actually good for certain types of workloads but it won't reach that that high TPS so six hours so I wouldn't say sweet spot in that regard”

0.60

Cross-cloud latency is not guaranteed by any provider, so setting up HTC Grid to automatically distribute work across multiple cloud providers in a latency-sensitive manner is not a recommended deployment pattern

normativehigh valueestablishednovelty 0/4durability 4/4· Dore Goï

“I have yeah and then question it would but that would not be a setup I would recommend anybody to do because the cross Cloud latency is not something anybody can guarantee okay”

0.53

HTC Grid's design principle 'all compute is Lambda' means that everything within HTC Grid implements or runs under the Lambda API, which abstracts computation as a function taking input and producing output, allowing customers to plug in any compute engine (different QR libraries, programming languages) without changing how they interact with the system

definitionhigh valuespeaker onlynovelty 2/4durability 4/4· Dore Goï

“all compute is a Lambda now this does not mean that it only uses AWS Lambda as a service to run this however AWS Lambda when it started as the whole function as a service kind of ecosystem had has a very goodly documented API and it actually basically takes two things the function you're going to run and the input you're given it and then it gives you an output you don't necessarily need to care about how does the API work how is it implemented those are the internals of the actual function that you give it it just takes an input and gives you an output so because of this we actually wanted to simplify the way you run all compute that is within ACC grid so everything that you plug in actually implements or runs under the compute that is a Lambda API”

0.52

Financial services workloads exhibit a bimodal task duration distribution where 40% of tasks complete in less than 2 seconds while the total task volume exceeds 200 million tasks per day for a single institution, requiring schedulers capable of processing 30,000+ tasks per second rather than typical 2,000-3,000 tasks per second

factualhigh valuespeaker onlynovelty 2/4durability 3/4· Carol Ivanov

“the most of the tasks are very short so in fact 40% of these tasks in this particular sample have been completed in less than two seconds and the second Insight here is that the total number of the tasks are very very large so we're talking about more than 200 millions of task per day for this particular uh financial institution now why this is important the the thing is if you want to process such large volume of short running tasks then your scheduler also should be able to uh move this volume of task in very efficient manner uh today um let's say a common schedule on average can process around maybe 2 to 3,000 task per second now in the contrast uh the HTC grid can work at the order of 30,000 per second”

0.52

HTC Grid was donated to Finos in order to obtain collaboration from the broader community and build a cloud-agnostic open-source solution

factualhigh valuespeaker onlynovelty 1/4durability 4/4· Carol Ivanov

“now this year we have donated HTC GD to fenos uh with our main objective just to uh get collaboration from the broader community and to uh build a cloud agnostic open source solution”

0.48

HTC Grid is a reusable blueprint providing a reference architecture for modern container-based solutions that uses common cloud services and incorporates best practices from multiple customer engagements and prototyping

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Carol Ivanov

“we have built HTC GD uh which is a reusable blueprint that provides a reference architecture for modern container based Solutions it uses common cloud services and addresses the problems above now HTC grid is the result of a multiple engagements multiple prototyping engagement we had with our customers and it incorporates our experience and best practices”

0.48

The main benefits financial institutions see from using HTC Grid are: cost savings by removing third-party licensing fees, simplified infrastructure and management, operational simplicity and reliability through fully managed services and serverless components, and accelerated adoption by customers

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Carol Ivanov

“the main benefits of the solution is what uh financial institution seen by using this blueprint is uh it's a cost saving because it's red removes all the third-party licensing fees and at the same same time it simplifies the infrastructure and the management of the solution uh it also provide operational Simplicity and reliability because a lot of this architecture is based on the fully managed services and serverless components and as a result uh we have seen speed up in adoption by customers of the solution”

0.48

The same logic for choosing data plane storage applies to the compute plane: EKS or Kubernetes is perhaps the most common choice because of interoperability and ability to run on-premises and on other cloud providers, but there is no one-size-fits-all solution and the list can be extended if customers want something else

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Carol Ivanov

“pretty much the same logic goes for the compute plane so uh uh eks the the kubernetes like would be perhaps the the the most common choice because of interoperability and ability to move to run it you know on PR and other Cloud providers Etc but again there's no like like one size fits all it depends and uh it's also possible to extend this list uh because again as for was saying the overall solution is amount of quat is really very small if somebody wants to have something else is quite possible”

0.48

HTC Grid supports custom and on-demand compute isolation by allowing customers to deploy multiple independent copies of the blueprint for different teams without resource contention, avoiding the inefficiencies of monolithic shared grids where teams must compete for resources

factualhigh valuespeaker onlynovelty 1/4durability 3/4· Dore Goï

“if you want to build like uh one monolithic grid which is how we used to do it in the old days then everybody kind of shared resources on that and you had to like compete with other teams you had to make sure that you're not like running over or the other way around you actually had to pay for that computer that was always on so then you actually had to ensure that you're utilizing it even if it didn't meant that you're running stuff that is not really useful right in this case you actually don't have to do that obviously part of that is because it's Cloud native so you're only paying for what you use but also because you can deploy multiple copies of this blueprint CU it's really easy to deploy and manage then you're actually able to like have on demand classes or on demand like ecosystems of HC grid that different teams have in use individually without actually overstepping on each other”

0.46

HTC Grid's control plane is fully serverless and uses managed AWS services (Lambda, SQS, DynamoDB, etc.) for task scheduling, status checking, health checking, and auto-scaling; because each service scales individually and independently, the control plane theoretically scales infinitely.

factualhigh valuespeaker onlynovelty 1/4durability 4/4· Dore

“the control plane which is fully using a serverless and managed Services products so everything here is a serverless product and that is actually a very good thing because one it almost like theoretically scales infinitely because they all scale individually you don't have to provision service it's not static scaling”

0.46

HTC Grid uses a pull-type task distribution model where workers pull tasks from a queue rather than the scheduler pushing tasks to workers; this design means the scheduler is not responsible for verifying worker liveness, reducing coordination overhead.

definitionhigh valuespeaker onlynovelty 1/4durability 4/4· Dore

“this is a pull typee system so when you schedule something those tasks make it into a que and then workers pull from that Q to sort it's not a push so the schuer is actually not responsible for making sure that all of the workers are alive there's other components that do this out of the box”

0.44

HTC Grid achieves high throughput (30,000 tasks per second) and low latency (0.3 seconds round-trip scheduling time from task submission to completion minus compute time) which enables it to handle both short sub-2-second tasks and longer tasks in the same system

factualhigh valuespeaker onlynovelty 2/4durability 2/4· Dore Goï

“ATC grid as we have tested today it's not the limit by far we have seen handle 30,000 tasks per second so that's the actual throughput of the scheduler but also equally if you have a round trip time of like say a zero SOS the actual from scheduling to completion minus the compute time is 0.3 seconds 300 milliseconds so it's a very false system from a scheduling perspective but it also allows you to actually run different types of tasks from end of day to intraday to potentially real-time risk as well”

0.43

When choosing a data plane (storage layer) for HTC Grid, the selection depends on customer experience and workload objectives: for time-sensitive workloads, ElastiCache provides very low latency; for cost optimization with large data volumes, S3 is the best choice; for POSIX filesystem requirements, options like FSx are available

normativehigh valuespeaker onlynovelty 1/4durability 3/4· Carol Ivanov

“it depends on the customer so it's uh it based on the their experience right so what they already running on Prem what they are familiar with and uh it depends on the objective of their workload right so for example some workloads can be very uh time sensitive right in this case when it comes for example for the data plane it is something like elastic cash for last this is would be the ideal it's give like the very very small latency if you want to optimize to the cost and if you want to move a lot of data you know then S3 probably the best choice in this in this things if it is a if you need POS file system then uh there apx FASA Etc”

0.42

HTC Grid is created as a blueprint for building high-throughput computing solutions in the cloud based on requirements observed from financial services workloads, where HTC stands for 'high throughput computing'

definitionhigh valuespeaker onlynovelty 0/4durability 4/4· Carol Ivanov

“for these reasons uh we have created HTC grid based on the the requirements that we uh have observed and HTC stands for hrut computing and as I mentioned before it is the blueprint for building h uh hydr put Computing Solutions in the cloud”

0.39

Future planned scaling improvements for HTC Grid include adding Karpenter for faster compute plane scaling with diverse purchasing options, scaling based directly on queue depth metrics to the cluster, and implementing more aggressive scaling algorithms to reach the required scales faster

forecasthigh valuespeaker onlynovelty 1/4durability 2/4· Dore Goï

“on the scaling front uh Carpenter is one of the big things that we want to add...that will actually help with the scaling of the compute plane so that it can reach the scales faster but also diversify use different purchasing options Etc but also we want to like Scale based on the depth metrics of the Q and actually have them go directly to the claes so that the scaling is much faster”

0.39

The HTC Grid control plane codebase is approximately 4,000 lines of code, making it small enough that anyone can easily read and understand the implementation, and the codebase is available in an open-source repository for community contribution

factualhigh valuespeaker onlynovelty 1/4durability 2/4· Dore Goï

“the actual code base around the control plane is about 4,000 lines of code that's about it so anybody can easily read and understand what's going on”

0.39

HTC Grid has three main components: a client application (REST API entry point), a control plane (fully serverless and managed services), and a compute plane (with modular variants like EC2, EKS, or Kubernetes), plus a data plane for task submission and retrieval

definitionhigh valuespeaker onlynovelty 0/4durability 3/4· Dore Goï

“if you're familiar with HBC systems then this is fairly traditional in that sense that there is a um like a client application portion here there is a control plane compute and data plane so the client application is basically your entry point into the system so that's what you would use to call the apis it's basically a rest API...when that comes in ultimately it's going to the control plane which is fully using a serverless and managed Services products...then we have the compute plane and as you can see this is that modularity aspect that I spoke to earlier so you've got different flavors of the compute plane that you can use”

0.39

Workload requirements for HTC Grid, including CPU-to-memory ratios (e.g., 1:2 or 1:8 for Java), are defined when creating the deployment and are fairly static across grid systems regardless of asset class or compute library

factualhigh valuespeaker onlynovelty 0/4durability 3/4· Dore Goï

“the workload requirements on this type of workload today are fairly static so when you talk to any of grid systems in the world today they will differ between asset classes Etc but they will all have a requirement of say my QR Library needs one CPU to to of memory or if it's Java 1 to8 or 1 to 16 so because of this it's actually defined when you create the deployment”