1 Dropout
If a neuron drops out in a hidden layer and no one is there to weigh it, does it influence the prediction?
2 Softmax
What is the softmax of a one-category classifier?
3 Phase transitions
A novice asked, “When does the cat detector awaken in the network?”
The master replied, “Before it sees the first whisker, it already dreams of its shape.”
The master replied, “Before it sees the first whisker, it already dreams of its shape.”
4 Data-centric
The SGD optimization was running on the HPC, and two data scientists started an argument.
One said the model moved, the other said the data moved;
they argued back and forth but could not reach a conclusion.
The finance department said, “It is not the data that moves, it is not the model that moves; it is your funding that moves.”
The two data scientists were awestruck.
One said the model moved, the other said the data moved;
they argued back and forth but could not reach a conclusion.
The finance department said, “It is not the data that moves, it is not the model that moves; it is your funding that moves.”
The two data scientists were awestruck.
5 AGI
Jõshû asked Nansen, “What is AGI?”
“Ordinary mind is AGI,” Nansen replied.
“Shall I try to seek after it?” Jõshû asked.
“If you try for it, you will become separated from it,” responded Nansen.
“How can I know AGI unless I try for it?” persisted Jõshû.
Nansen said, “AGI is not a matter of knowing or not knowing.
Knowing is delusion; not knowing is confusion.
When you have really reached true AGI beyond doubt, you will find it as vast and boundless as outer space.
How can it be talked about on the level of right and wrong?”
With these words, Jõshû came to a sudden realization.
“Ordinary mind is AGI,” Nansen replied.
“Shall I try to seek after it?” Jõshû asked.
“If you try for it, you will become separated from it,” responded Nansen.
“How can I know AGI unless I try for it?” persisted Jõshû.
Nansen said, “AGI is not a matter of knowing or not knowing.
Knowing is delusion; not knowing is confusion.
When you have really reached true AGI beyond doubt, you will find it as vast and boundless as outer space.
How can it be talked about on the level of right and wrong?”
With these words, Jõshû came to a sudden realization.
6 Embedding Space
A student asked, “If every word lives in a high-dimensional realm, how do I find its true neighbour?”
The master pointed to two distant points and said, “Trust not the distance, but the direction.”
The master pointed to two distant points and said, “Trust not the distance, but the direction.”
7 Tech support
Yakusan the HPC support officer had not ascended the rostrum for a long time.
The institute steward said, “All the staff has been wishing for instruction for a long time. Please, Master, give the noobs a seminar.”
Yakusan had the BEL rung.
The noobs gathered.
Yakusan ascended the rostrum and sat there for a while.
Then he descended and returned to his office.
The institute steward followed him and asked, “You said a while ago that you would give the noobs a seminar. Why didn’t you speak even a word?”
Yakusan said, “For neural nets, there are neural net specialists; for physics simulations, there are physics simulation specialists.
Why do you have doubts about this old HPC support officer?”
The institute steward said, “All the staff has been wishing for instruction for a long time. Please, Master, give the noobs a seminar.”
Yakusan had the BEL rung.
The noobs gathered.
Yakusan ascended the rostrum and sat there for a while.
Then he descended and returned to his office.
The institute steward followed him and asked, “You said a while ago that you would give the noobs a seminar. Why didn’t you speak even a word?”
Yakusan said, “For neural nets, there are neural net specialists; for physics simulations, there are physics simulation specialists.
Why do you have doubts about this old HPC support officer?”
8 Blurry JPEGs
A student asked the sage, “How many books must one read to know everything that a language model can understand?”
The sage smiled and replied, “The library of knowledge is not bounded by pages but by space itself.”
The student pressed, “But if the library grows infinitely, how can I ever find a single truth?”
The sage pointed to the vast shelves and said, “Google it.”
The sage smiled and replied, “The library of knowledge is not bounded by pages but by space itself.”
The student pressed, “But if the library grows infinitely, how can I ever find a single truth?”
The sage pointed to the vast shelves and said, “Google it.”
9 Overfitting
A junior model boasted, “I know every detail of the training set.”
The master replied, “Then when you meet the unseen world, who will you be?”
The master replied, “Then when you meet the unseen world, who will you be?”
10 Reinforcement Learning
The trainee asked, “How do I know if my agent has learned?”
The master answered, “When it no longer seeks reward, but serves the task itself.”
The master answered, “When it no longer seeks reward, but serves the task itself.”
11 Developmental interpretability
The apprentice peered into the first epoch and cried, “All I see are blotches of colour!”
The mentor said, “Attend again at recess: the scribbles become letters.”
By lunchtime the apprentice gasped, “They spell worlds.”
The mentor said, “Attend again at recess: the scribbles become letters.”
By lunchtime the apprentice gasped, “They spell worlds.”
12 Sources
Blue Cliff Record, Odes to a Classic Hundred Standards
Gateless barrier