1
00:00:00,000 --> 00:00:02,998
So talk to me about trumpets.

2
00:00:03,898 --> 00:00:05,831
You tell the Brown M&Ms story, and.

3
00:00:05,831 --> 00:00:08,363
Okay, I'll, I'll play it with trumpets.

4
00:00:08,363 --> 00:00:11,495
Okay, so most, most people
probably know this story already.

5
00:00:11,762 --> 00:00:15,160
But the band Van Halen, famously

6
00:00:15,394 --> 00:00:19,559
in their concert
rider, had a specification.

7
00:00:19,559 --> 00:00:21,791
That concert rider is like some documents.

8
00:00:21,791 --> 00:00:22,657
It goes to the venue.

9
00:00:22,657 --> 00:00:23,557
Exactly. Yeah.

10
00:00:23,557 --> 00:00:24,890
That's like all of the

11
00:00:24,890 --> 00:00:29,755
the the requests and demands of the band
has for supporting them at the venue.

12
00:00:29,788 --> 00:00:30,521
Okay.

13
00:00:30,521 --> 00:00:34,386
So they had this, you know, rider
that says that in their green room,

14
00:00:34,453 --> 00:00:36,352
there must be.

15
00:00:36,352 --> 00:00:38,318
A the green room is like the place

16
00:00:38,318 --> 00:00:40,351
where they hang out before
and after the show, right?

17
00:00:40,351 --> 00:00:42,250
Like get ready. Okay.

18
00:00:42,250 --> 00:00:43,549
It might not have been the green room,

19
00:00:43,549 --> 00:00:45,915
but anyway,
there has to be this bowl of M&Ms.

20
00:00:45,915 --> 00:00:50,447
But it has to have all of
the, the brown M&Ms removed.

21
00:00:51,580 --> 00:00:54,578
No poop candy.

22
00:00:57,711 --> 00:00:58,310
And if you

23
00:00:58,310 --> 00:01:01,309
haven't heard this story before,
you know, for,

24
00:01:01,542 --> 00:01:05,541
it used to be, like,
laughed at as an example of, bands

25
00:01:05,574 --> 00:01:06,707
being very precious,

26
00:01:06,707 --> 00:01:10,372
but it turns out that they actually put
that in there, for a reason,

27
00:01:10,539 --> 00:01:13,804
which was they knew, if,
you know, if they came in and

28
00:01:13,804 --> 00:01:17,603
there was a bowl of M&Ms, that was that,
that had brown Eminem's in it,

29
00:01:17,870 --> 00:01:21,801
that somebody had not gone
over the writer, with a fine tooth comb.

30
00:01:22,734 --> 00:01:25,000
And that meant that they needed to check

31
00:01:25,000 --> 00:01:28,199
all of, like, the, the

32
00:01:29,398 --> 00:01:29,798
all the,

33
00:01:29,798 --> 00:01:33,563
you know, the lighting more carefully
and the electrical and, like, the weight,

34
00:01:33,963 --> 00:01:39,528
you know, the weight capacity
of the of the stage and things like that.

35
00:01:39,761 --> 00:01:40,927
Because if they didn't,

36
00:01:40,927 --> 00:01:44,193
if they didn't observe the Eminem part,
they probably missed something else.

37
00:01:45,192 --> 00:01:47,558
So really, it's a did you read this right?

38
00:01:47,558 --> 00:01:51,157
Which is exactly what we need
for our lives.

39
00:01:51,590 --> 00:01:52,023
Right?

40
00:01:52,023 --> 00:01:53,256
I mean, it's not just did you read this?

41
00:01:53,256 --> 00:01:54,989
I believe that you read this.

42
00:01:54,989 --> 00:01:57,987
I could check that in the tool call,
but do you remember it?

43
00:01:58,321 --> 00:02:00,020
Is it still in your head? Right.

44
00:02:00,020 --> 00:02:01,353
Because what we're doing,

45
00:02:01,353 --> 00:02:04,651
at least at this juncture, a lot of what
we're doing is the humans in the loop

46
00:02:05,118 --> 00:02:06,817
is attention management. It seems like.

47
00:02:06,817 --> 00:02:07,950
Oh, absolutely.

48
00:02:07,950 --> 00:02:09,383
Same thing we do with each other.

49
00:02:09,383 --> 00:02:12,382
Yeah. And our children and ourselves.

50
00:02:12,915 --> 00:02:13,848
But but yeah.

51
00:02:13,848 --> 00:02:18,713
Because, 
the example here is I have a task

52
00:02:18,713 --> 00:02:23,011
implementation process
in the claude.md for the project.

53
00:02:23,378 --> 00:02:26,743
And it says
as part of implementing a task,

54
00:02:27,110 --> 00:02:31,608
you should gather context and check
with all the capability owners

55
00:02:31,608 --> 00:02:32,407
while you're doing that.

56
00:02:33,507 --> 00:02:34,940
And then,

57
00:02:34,940 --> 00:02:39,172
you should, I don't know.

58
00:02:39,172 --> 00:02:40,638
Oh, decide how you're going to test it.

59
00:02:40,638 --> 00:02:44,669
There's like, verification check decide
how you're going to observe it.

60
00:02:44,936 --> 00:02:46,602
Write that out in a plan.

61
00:02:46,602 --> 00:02:48,901
Okay. Then start implementing the task.

62
00:02:48,901 --> 00:02:50,401
Then do the verification

63
00:02:50,401 --> 00:02:53,399
check with the capability owners again,
blah blah blah blah blah.

64
00:02:54,232 --> 00:02:57,098
And at the end
the last step is print a trumpet

65
00:02:57,098 --> 00:03:00,097
in Ascii art.

66
00:03:00,497 --> 00:03:01,863
Print a

67
00:03:01,863 --> 00:03:02,563
trumpet. Print a trumpet.

68
00:03:02,563 --> 00:03:07,027
And often I'm
pretty sure that whole process

69
00:03:08,394 --> 00:03:09,527
gets out of its head.

70
00:03:09,527 --> 00:03:12,525
It's too far up in the context,
or it's done

71
00:03:12,525 --> 00:03:15,924
the thing that it does automatically
of condensing the context.

72
00:03:15,924 --> 00:03:20,389
So far, and it's lost that list.

73
00:03:20,389 --> 00:03:23,454
So if it doesn't print a trumpet,
then maybe it didn't

74
00:03:23,454 --> 00:03:25,420
check with the capability owners.

75
00:03:25,420 --> 00:03:27,686
And then once in a while

76
00:03:27,686 --> 00:03:32,284
I come back to my desk and I'm like,
what the heck is that?

77
00:03:32,284 --> 00:03:35,716
Oh, that's it's
attempted to trumpet, right?

78
00:03:35,750 --> 00:03:37,782
I told her to build a trumpet.

79
00:03:37,782 --> 00:03:40,381
I'm not super good
at checking for the trumpet, but,

80
00:03:41,381 --> 00:03:43,613
I call
the downfall of attention management.

81
00:03:43,613 --> 00:03:46,579
All right, I that's.

82
00:03:46,579 --> 00:03:49,578
Yeah, well,
this is the difference between, like,

83
00:03:49,744 --> 00:03:54,676
being responsible for a task
and doing a task, like, as a as a manager.

84
00:03:54,676 --> 00:03:57,741
It's my job to make sure
this stuff gets done, not to do it.

85
00:03:57,741 --> 00:03:58,674
But I'm the person

86
00:03:58,674 --> 00:04:01,673
who has to come back and check
and make sure it didn't get dropped.

87
00:04:01,673 --> 00:04:02,606
Right.

88
00:04:02,606 --> 00:04:06,305
And so me managing Claude
I need to check for the trumpet.

89
00:04:06,305 --> 00:04:07,404
Do I check for the trumpet. No.

90
00:04:07,404 --> 00:04:11,003
But every time it tries to draw a trumpet
it makes me happy.

91
00:04:13,569 --> 00:04:16,667
It's really bad at drawing ASCII trumpets
actually?

92
00:04:16,967 --> 00:04:19,300
I was like maybe it’s
because its context is full.

93
00:04:19,300 --> 00:04:24,231
So this morning we asked it to to draw
a trumpet like empty context just right.

94
00:04:24,264 --> 00:04:26,763
No, it just can't draw trumpets.

95
00:04:27,963 --> 00:04:29,562
There's many things it could draw
really well.

96
00:04:29,562 --> 00:04:31,928
It trumpets. No, one of them.

97
00:04:31,928 --> 00:04:35,027
So has it been working like like
do you find that

98
00:04:35,027 --> 00:04:38,026
it hasn't missed the other steps
when it does draw the trumpet?

99
00:04:39,125 --> 00:04:41,758
I didn't check.

100
00:04:41,758 --> 00:04:45,023
Because I mean, you know,
there's the thing with human attention.

101
00:04:45,023 --> 00:04:46,522
There's the thing where we mostly pay

102
00:04:46,522 --> 00:04:49,521
attention to the first thing we heard
and the last thing we heard.

103
00:04:49,655 --> 00:04:51,554
And it has the same property, right? Yeah.

104
00:04:51,554 --> 00:04:56,385
And so I guess I would wonder whether,
you know, it actually does the end.

105
00:04:56,452 --> 00:04:58,851
It always does the in between things
or if it pays

106
00:04:58,851 --> 00:05:00,317
the most attention to the trumpet.

107
00:05:00,317 --> 00:05:03,316
Now see this is, this is where

108
00:05:03,449 --> 00:05:07,814
I want to code my own agent
so that I can explore this kind of thing.

109
00:05:08,014 --> 00:05:12,046
And because that kind of check,
if you know what your process is

110
00:05:12,446 --> 00:05:16,211
and you put that implement that process
in deterministic code as well.

111
00:05:16,211 --> 00:05:17,011
Yeah.

112
00:05:17,011 --> 00:05:20,943
Then you can either
force it to check or you can

113
00:05:22,042 --> 00:05:22,509
check that it

114
00:05:22,509 --> 00:05:25,507
checked by, by looking at its tools.

115
00:05:26,474 --> 00:05:28,173
Yeah. I I mean.

116
00:05:28,173 --> 00:05:31,072
We have evals
for this actually at honeycomb

117
00:05:31,072 --> 00:05:35,004
we have evals that run an agent

118
00:05:35,204 --> 00:05:38,602
and they're looking for outcomes
and some of the outcomes they look for.

119
00:05:38,602 --> 00:05:40,602
Did you call this tool.

120
00:05:40,602 --> 00:05:42,401
When you say an email.

121
00:05:42,401 --> 00:05:45,033
Okay. Eval evaluation.

122
00:05:45,033 --> 00:05:47,532
This is what we have

123
00:05:47,532 --> 00:05:50,931
instead of tests for LLMs
because you can't.

124
00:05:51,264 --> 00:05:53,630
We being honeycomb or more widely.

125
00:05:53,630 --> 00:05:55,229
Oh much more widely okay.

126
00:05:55,229 --> 00:05:55,829
Right. Right.

127
00:05:55,829 --> 00:05:58,828
There's there's evals at the model level.

128
00:05:59,261 --> 00:06:01,960
Which are the most common
ones of for this prompt.

129
00:06:01,960 --> 00:06:03,893
And this model.

130
00:06:03,893 --> 00:06:07,691
And here were some input from the user,
like the question or whatever.

131
00:06:08,424 --> 00:06:11,123
Here's the output. Good or bad.

132
00:06:11,123 --> 00:06:14,455
And you can do that on production data.

133
00:06:14,722 --> 00:06:18,587
And then you have to like come up
with some heuristic for determining good

134
00:06:18,587 --> 00:06:19,820
or bad.

135
00:06:19,820 --> 00:06:21,786
Also you can get the user to thumb up
thumbs down.

136
00:06:21,786 --> 00:06:23,618
That's another, another one. Yeah.

137
00:06:23,618 --> 00:06:27,983
But you can also do this on like a
golden master, which is what our evals are

138
00:06:28,583 --> 00:06:32,415
there at the agent level,
but still, for this input,

139
00:06:33,415 --> 00:06:36,413
we expect this results.

140
00:06:36,413 --> 00:06:38,479
And in our case, our heuristics are like.

141
00:06:38,479 --> 00:06:41,078
The result includes
this bit of texture, right?

142
00:06:41,078 --> 00:06:42,245
It got the number right?

143
00:06:42,245 --> 00:06:45,043
Yeah. It ran a tool call.

144
00:06:45,043 --> 00:06:48,642
You know, it ran this particular query
that looks kind of like this

145
00:06:48,642 --> 00:06:51,641
because otherwise it's doing it wrong.

146
00:06:52,007 --> 00:06:55,206
So you can have all kinds of heuristics
about the output,

147
00:06:55,573 --> 00:06:58,572
but evals are,

148
00:06:58,638 --> 00:07:01,637
they're not property based because
we're not randomly generating input.

149
00:07:02,170 --> 00:07:06,069
But they are property checkers.

150
00:07:06,069 --> 00:07:08,701
Because we're not saying is the output.

151
00:07:08,701 --> 00:07:09,201
Exactly.

152
00:07:09,201 --> 00:07:13,699
This we're saying is the kind of like
this doesn't include this.

153
00:07:13,866 --> 00:07:17,498
You can use LLM as judge and say,
how correct was this

154
00:07:17,997 --> 00:07:20,663
to a completely different model,
or the same model with a different

155
00:07:20,663 --> 00:07:21,763
summoning spell?

156
00:07:22,929 --> 00:07:23,795
Yeah.

157
00:07:23,795 --> 00:07:26,827
So evals are how we evaluate our agents
and models.

158
00:07:26,827 --> 00:07:28,260
Okay. Yeah.

159
00:07:28,260 --> 00:07:31,959
I mean, the thing that I was doing for
a while, is the different

160
00:07:32,492 --> 00:07:33,158
a little different.

161
00:07:33,158 --> 00:07:34,424
It's not evaluation.

162
00:07:34,424 --> 00:07:36,190
It's really just like guardrails.

163
00:07:36,190 --> 00:07:40,522
But but, you know, I found that
a lot of times they would forget things,

164
00:07:40,589 --> 00:07:43,421
or it would keep trying to do things
even though,

165
00:07:43,421 --> 00:07:45,487
like,
you know, I'd put guidance not to do that.

166
00:07:45,487 --> 00:07:49,019
There was some command that it was getting
subtly wrong every single time.

167
00:07:49,285 --> 00:07:52,284
Almost every time,
even if I put, you know,

168
00:07:52,617 --> 00:07:54,383
even if I put guidance

169
00:07:54,383 --> 00:07:58,915
in, it's in its rules and I wound up...

170
00:07:58,915 --> 00:08:02,180
Oh, like mine likes to run Python
instead of Python three.

171
00:08:02,214 --> 00:08:05,113
Right. That kind of thing. Yeah.
And so I wound up like creating.

172
00:08:05,113 --> 00:08:05,413
Oh yeah.

173
00:08:05,413 --> 00:08:08,945
There was some command that it kept
trying to run and I was like, fine,

174
00:08:08,945 --> 00:08:12,143
I'm just going to install the library
and then it's going to work.

175
00:08:12,443 --> 00:08:13,576
Oh, I'm meaner than that.

176
00:08:13,576 --> 00:08:15,742
I like more of a so.

177
00:08:15,742 --> 00:08:17,708
I actually highly efficient way to do it.

178
00:08:17,708 --> 00:08:18,541
Wrote the system.

179
00:08:18,541 --> 00:08:21,973
No, I just put a, a, stub for

180
00:08:21,973 --> 00:08:24,972
whatever command that it was getting wrong
because it needed to use the right one.

181
00:08:25,105 --> 00:08:28,104
That just yelled at it
whenever I called the stub.

182
00:08:28,570 --> 00:08:30,603
Oh, I just made the stub.

183
00:08:30,603 --> 00:08:33,235
I just made what it wanted to type work.

184
00:08:33,235 --> 00:08:34,568
That is also an approach.

185
00:08:34,568 --> 00:08:37,367
I have this approach
with things in my house.

186
00:08:38,666 --> 00:08:39,599
When I can't find

187
00:08:39,599 --> 00:08:43,798
something, like
there was a lid to a pan the other day.

188
00:08:43,798 --> 00:08:46,663
Where is this
lid, is it where the pan is. No.

189
00:08:46,663 --> 00:08:49,662
Is it in this place of flat things.

190
00:08:49,662 --> 00:08:52,894
No. Is it in this other place of plastic
things. Yes.

191
00:08:53,661 --> 00:08:53,994
Okay.

192
00:08:53,994 --> 00:08:55,460
And then when I put it away,
I put it in the first place.

193
00:08:55,460 --> 00:08:58,292
I looked.

194
00:08:58,292 --> 00:09:01,358
Because next time it'll be, I'll probably
look at the same place again.

195
00:09:01,358 --> 00:09:05,756
And so that that's my I installed
its library that it wanted to have.

196
00:09:05,923 --> 00:09:08,022
It didn't it was.

197
00:09:08,022 --> 00:09:10,454
Oh yeah. It was like a timer, a timeout.

198
00:09:10,454 --> 00:09:11,887
It was timeout.

199
00:09:11,887 --> 00:09:12,353
Oh, okay.

200
00:09:12,353 --> 00:09:15,352
It wanted to run timeout. Blah. Command.

201
00:09:15,952 --> 00:09:18,951
Which setting a timeout
is a good thing for it to do.

202
00:09:19,117 --> 00:09:21,383
But that command
didn't work on my system. And

203
00:09:22,416 --> 00:09:23,882
so I just installed it.

204
00:09:23,882 --> 00:09:24,849
It was fine.

205
00:09:24,849 --> 00:09:28,814
So sometimes we just change the world
to make it more compatible with whoever

206
00:09:28,814 --> 00:09:30,380
we're working with. Right.

207
00:09:30,380 --> 00:09:34,112
Which you know, I mean,
what that effectively is with, with these

208
00:09:34,112 --> 00:09:38,543
things is, is changing our environment
to be more like the baseline of the world.

209
00:09:38,543 --> 00:09:39,310
Yeah. You know, of.

210
00:09:39,310 --> 00:09:39,610
Yeah.

211
00:09:39,610 --> 00:09:42,242
The consensus tools
because generally they're,

212
00:09:42,242 --> 00:09:44,741
you know, they're trained on what,
what people are using.

213
00:09:44,741 --> 00:09:47,240
Right. The average of the expectations
of the internet.

214
00:09:47,240 --> 00:09:47,740
Yeah.

215
00:09:47,740 --> 00:09:50,472
And I am for that kind of compatibility.

216
00:09:50,472 --> 00:09:51,705
Yeah.

217
00:09:51,705 --> 00:09:55,337
And the brown M&Ms were there to see

218
00:09:55,337 --> 00:09:58,902
whether you were noticing the things
that you do need to do differently.

219
00:09:58,902 --> 00:10:00,102
Right.

220
00:10:00,102 --> 00:10:02,801
That are not just what you assume
every band wants.

221
00:10:02,801 --> 00:10:04,067
Right. Exactly.

222
00:10:04,067 --> 00:10:04,233
Yeah.

223
00:10:04,233 --> 00:10:09,431
So you can leave
brown M&M clauses in your claude.md Yes.

224
00:10:10,964 --> 00:10:12,430
And You've just

225
00:10:12,430 --> 00:10:15,496
been listening
to Machines of Code and Grace.

226
00:10:15,562 --> 00:10:17,428
I'm Jesse Tron, and I'm.

227
00:10:17,428 --> 00:10:20,660
And this is a podcast
where we have discussions about the joys

228
00:10:20,660 --> 00:10:24,459
and the challenges of being a full circle

229
00:10:24,459 --> 00:10:27,458
developer,
because our job isn't writing code.

230
00:10:27,524 --> 00:10:30,623
Our job is building and improving
feedback loops.

231
00:10:30,623 --> 00:10:35,588
Those loops include our code
and our coding agents and our teams and

232
00:10:35,621 --> 00:10:40,819
operations, and our users and the larger
communities that we work in.

233
00:10:40,853 --> 00:10:42,219
Like this one.

234
00:10:42,219 --> 00:10:46,117
That's why each episode we bring
you ten minutes of discussion

235
00:10:46,117 --> 00:10:49,116
about what
we're learning from our work right now.

236
00:10:49,116 --> 00:10:51,449
But the conversation
doesn't have to stop there.

237
00:10:51,449 --> 00:10:52,915
If you like what you just heard,

238
00:10:52,915 --> 00:10:57,180
why not head over to graceful
dev slash machines?

239
00:10:57,213 --> 00:11:00,145
Sign up for our code and Grace newsletter.

240
00:11:00,145 --> 00:11:04,310
Yeah, we'll send you some key takeaways
from our recent episodes.

241
00:11:04,310 --> 00:11:06,143
Hand-selected by humans.

242
00:11:06,143 --> 00:11:06,843
That's us.

243
00:11:06,843 --> 00:11:08,109
And along with that,

244
00:11:08,109 --> 00:11:11,974
we'll send you some news,
some links to some of our favorite things.

245
00:11:12,007 --> 00:11:13,507
Again, that's graceful.

246
00:11:15,673 --> 00:11:17,139
What else will they get?

247
00:11:17,139 --> 00:11:18,272
I'm glad you asked.

248
00:11:18,272 --> 00:11:21,470
Jesus, because we'll also send you

249
00:11:21,470 --> 00:11:25,569
an extra good discount
for a pro subscription to Graceful Dev.

250
00:11:25,902 --> 00:11:28,601
That's where you can get access
to hundreds of videos

251
00:11:28,601 --> 00:11:31,966
and dozens of courses
for growing your programing practice.

252
00:11:32,466 --> 00:11:35,098
Also, before you go,
if you like what you heard,

253
00:11:35,098 --> 00:11:36,531
why not subscribe to the podcast

254
00:11:36,531 --> 00:11:40,663
so you can join us
next time on Machines of Code and Grace.

255
00:11:40,696 --> 00:11:41,629
See you soon.