Rendered at 13:00:51 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
jhgg 20 hours ago [-]
This blog post spends a lot of time telling why gang scheduling is needed here (a fact obvious to anyone using the nvl72s), but doesn't actually go into detail about how their scheduler works.
At work, we have built something very similar, a custom topology aware scheduler for workloads, and the most interesting things about such a project are entirely absent from this article, making it more of a fluff/marketing piece than an actually interesting technical blog in this area.
Our next iteration of this scheduler aims to scale to O(millions) of GPUs, with a scheduling cycle time measured in the milliseconds.
ismailmaj 20 hours ago [-]
Feels AI-generated, so much text real estate is spent explaining the basics in sufficient detail then just two paragraphs on their scheduler with a lot of hand-waving.
zhwu 20 hours ago [-]
This blog's goal is to introduce the problem around the new NVL72 system with GPU failures in production, which not all people outside of xAI are aware of. We have more blogs coming up for what we built to handle topology-aware scheduling, defragmentation and hot swapping during failures. The short version of the scheduling piece: jobs are decomposed into 16-node slices placed atomically on a single rack, with two spares per rack reserved for hot swaps.
Seems very relevant to what you are building. Quoting what Ali said: "Quick PoC?"
covi 17 hours ago [-]
Why do you target millisecond-level cycle time? Are you looking at very fine-grained tasks like elastic sandboxes, in addition to beefy pre- and post-training jobs?
zamalek 18 hours ago [-]
> O(millions)
Big-O doesn't include constant factors for a reason (never mind just being a constant entirely), but maybe I am misunderstanding what you are trying to say?
arjie 17 hours ago [-]
It's not Big-O complexity notation. It means "a few millions". It's sort of how the word "sanction" can mean one thing or a different thing. The textual glyph sequence means a different thing based on context in English. Pretty normal.
I like to call it Big-O Magnitude Notation.
alienbaby 14 hours ago [-]
why not just say 'millions'
"Our next iteration of this scheduler aims to scale to millions of GPUs"
arjie 14 hours ago [-]
I treat it as more akin to the (slightly more-awkward) "single-digit millions" whereas "millions" could go over. It's just a SV-ism. I usually find it pretty easy to disambiguate and I confuse it as little as I do the same / in dy/dx vs. 1/2. Notational overload is pretty normal in human communication, I think.
jhgg 18 hours ago [-]
"Order of millions of GPUs being scheduled on" - which is to say when a workload is submitted to the scheduler, it makes a scheduling decision and the individual compute nodes are being programmed within milliseconds.
This means that if say a job requesting GPUs is submitted with a given topology constraint, we would already computed the physical nodes that satisfy the topology requirement, and begun preemption / begun spinning up processes on the target machines within a few milliseconds of the job being submitted. If there were no preemptions required (which means we need to first wait for graceful termination), the job is fully scheduled in the blink of an eye, whether it's on tens or hundreds of thousands of GPUs.
deadnomore 18 hours ago [-]
You are abusing big O which is worst case time complexity scaling. If I simplify your notation, then you just wrote O(1)
jhgg 18 hours ago [-]
Should have said "order of"
28 minutes ago [-]
nextaccountic 14 hours ago [-]
It means ~millions
porridgeraisin 20 hours ago [-]
> making it more of a fluff/marketing piece than an actually interesting technical blog in this area.
/genuine do you recommend any other blog?
jeffinhat 17 hours ago [-]
open source the scheduler
retiredpapaya 17 hours ago [-]
jhgg and jlf PLEASE OPEN SOURCE THE SCHEDULER
i suspect you also have some great 60Hz cluster observability software ;)
Yes, SkyPilot Platform built the scheduler on top of Kueue. There is some shortcoming of Kueue's TAS that you have to resolve, e.g., you cannot do TAS with a Elastic job, which makes the hot swapping not possible for large-scale jobs on NVL72 racks: https://kueue.sigs.k8s.io/docs/concepts/elastic_workload/#li...
"No Topology-Aware Scheduling (TAS) support."
There is another blog talking about this coming soon.
timzaman 17 hours ago [-]
Who has nvl72s that this isnt absolutely plainly obvious to - and has been for years. Not just that - topology has -always- been relevant to scheduling, and hpc schedulers have always been aware. What a weird blogpost.
novalite 19 hours ago [-]
This reads a bit disingenuous and click-baity. There's no "new" scheduler. Gang scheduling and rack-aware scheduling have been around since the days of HTCondor, and are put in practice at almost every large GPU deployment I've seen. The net new information added by this blog is close to zero, and feels like shilling for SkyPilot.
hinkley 19 hours ago [-]
The danger of being excited about old techniques is giving the impression that you are years behind everyone else.
A coworker once said of a particular breed of manager who is an emphatic late adopter: it’s as if every time he goes to the dentist he reads an old trade journal in the waiting room and comes back to work with a “new” idea that everyone else is already busy replacing.
Are there any actual newer techniques? For instance, putting 11% of your hardware on reserve for hardware failures seems a bit generous to me for such an expensive cluster. Even if it’s down from some much larger number. One solution to this I’ve seen for handling cpu and memory variability with bin packing is to pool the spare capacity between tasks. For instance a data center with four zones might have 7 backup generators, instead of eight or twelve, counting on clusters of generator failures not to happen frequently.
And of course there’s always running low priority tasks on the spare hardware and evicting them on a fault. However those low priority tasks tend to develop their own shadow priorities when they are in house. But that’s a management problem not a technical one.
zan2215 19 hours ago [-]
"Not all high-speed interconnects are created equal"
I think he's a red rising fan
At work, we have built something very similar, a custom topology aware scheduler for workloads, and the most interesting things about such a project are entirely absent from this article, making it more of a fluff/marketing piece than an actually interesting technical blog in this area.
Our next iteration of this scheduler aims to scale to O(millions) of GPUs, with a scheduling cycle time measured in the milliseconds.
Seems very relevant to what you are building. Quoting what Ali said: "Quick PoC?"
Big-O doesn't include constant factors for a reason (never mind just being a constant entirely), but maybe I am misunderstanding what you are trying to say?
I like to call it Big-O Magnitude Notation.
"Our next iteration of this scheduler aims to scale to millions of GPUs"
This means that if say a job requesting GPUs is submitted with a given topology constraint, we would already computed the physical nodes that satisfy the topology requirement, and begun preemption / begun spinning up processes on the target machines within a few milliseconds of the job being submitted. If there were no preemptions required (which means we need to first wait for graceful termination), the job is fully scheduled in the blink of an eye, whether it's on tens or hundreds of thousands of GPUs.
/genuine do you recommend any other blog?
i suspect you also have some great 60Hz cluster observability software ;)
its good for the world if you do it
"No Topology-Aware Scheduling (TAS) support."
There is another blog talking about this coming soon.
A coworker once said of a particular breed of manager who is an emphatic late adopter: it’s as if every time he goes to the dentist he reads an old trade journal in the waiting room and comes back to work with a “new” idea that everyone else is already busy replacing.
Are there any actual newer techniques? For instance, putting 11% of your hardware on reserve for hardware failures seems a bit generous to me for such an expensive cluster. Even if it’s down from some much larger number. One solution to this I’ve seen for handling cpu and memory variability with bin packing is to pool the spare capacity between tasks. For instance a data center with four zones might have 7 backup generators, instead of eight or twelve, counting on clusters of generator failures not to happen frequently.
And of course there’s always running low priority tasks on the spare hardware and evicting them on a fault. However those low priority tasks tend to develop their own shadow priorities when they are in house. But that’s a management problem not a technical one.