Scaling exercises

✅ Do come up with a quick plan for each exercise before starting

$\underset{―}{Why?}$‍ It’s easier to scale if you’ve started with a quick notion of the concrete concept, types you’ll use, and how many questions the final exercise will have

Example. “This exercise will focus on comparing mitosis with meiosis, assuming students have already learned the fundamentals of each. It will have 4 types based on the four key similarities and differences, starting with 2 Ur questions in each of the 4 types (8 total), that will scale x3 (for a total of 24).”

✅ Do write your first questions so they are conducive to scaling.

$\underset{―}{Why?}$‍ It’s easier to scale if you already have a base that is designed with scaling in mind. It also allows for greater collaboration if you can hand off scale-able questions to other team members who can focus exclusively on scaling.

Example. “What would be the primary expected outcome to writing to a Senator?” is difficult to scale and feel unique. “Sally wants to write to her Senator. What is her primary expected outcome?” can scale in both learning objective and uniqueness to “Robert wants to write to the President. What is his primary expected outcome?”

Example 1. A graphing question on SRAS may scale quickly to a definition question about what SRAS is. Scaling from definition to a graphing question is more complex.

Example 2. Scaling from a passage-based question on a Supreme Court case is easier to scale to a scenario question, and then to a definition question, because the hardest part (finding a good passage) is already done.

🚫 Don’t continue scaling once students can game the system by noticing a pattern in the problem.

$\underset{―}{Why?}$‍ If a student starts to recognize a pattern in the contextual situation of the question, they can start to “game” the questions by following that pattern, rather than applying the skill regardless of the particular situation.

✅ Do use types to distinguish concrete sub-concepts or varieties in an exercise.

$\underset{―}{Why?}$‍ Types are the purple categories that are used to divide up questions in an exercise. If a student gets questions from type 1, 2, and 3 right, but they get sub-concept/variety in type 4 wrong, they should be able to try again and see the same variety from their first practice and isolate their learning on improving type 4. Without using types, on a second attempt a Do-4 may pull randomly only from varities 1-3.

✅ Do aim for diversity across types more than diversity within types.

$\underset{―}{Why?}$‍ On their 2nd+ time through the exercise, students should be able to focus their energy on improving their understanding of concepts behind the questions they got wrong, while reinforcing those behind the questions they answered correctly. Serving them a set of questions that are recognizable as different from the first set they saw, but that are on the whole substantially similar, can reduce the overall cognitive load and potential for frustration.

An added benefit to this approach is streamlined exercise creation! Designing a few diverse questions (each one as a type) and then a few permutations of each of those (items to fill up the types) is often an efficient way create exercises.

Example 1. In a GoPo course, the four types may assess the same learning objective, at the same difficulty, using a primary source (type 1), a secondary source (type 2), a political cartoon (type 3), and a graph (type 4). If they are roughly the same difficulty, students have diversity in seeing that learning objective presented in four different ways. The scaled types for primary source may use that same source and slightly vary the stem and options. As such, there will be noticable diversity in a user experience in the first Do-N (seeing different passages, cartoons, etc), but in the second Do-N they may encounter similar stimuli and have the opportunity to re-engage with them in the context of a new stem or options.

Example 2. The following are two of four types appearing in an upcoming AP Bio skill. In that skill, each type contains three items. Only the stimulus and stem are shown here, although note that changes to options is also a great way to control diversity within types.

🚫 Don’t put very different skills or concepts in an exercise.

$\underset{―}{Why?}$‍ Students should be able to engage with a focused skill or concept in an exercise. If there are more than one, they may not be able to isolate which of the 2+ skills they are struggling with and be able to get targeted practice in that area.

NOTE: this practice may not be feasible or appropriate for a given course, particularly early iterations - see below: ‘Do consider the right concept/skill granularity for exercises’

Example. A student may understand mitosis, but not know how to apply that knowledge in the concept of a graph. If in a single exercise a student student get one question wrong (mitosis + graph) and another right (mitosis + diagram), they may keep practicing thinking they don’t understand mitosis, when in reality their weakness may be graphs or mitosis-in-the-context-of-graphs. If the latter is so fundamental to the learning objectives of the course, and students will struggle without mastering that essential, then we should have discrete exercises for that targeted practice.

✅ Do consider the right concept/skill granularity for exercises.

$\underset{―}{Why?}$‍ We want exercises to be as atomized as possible, so students can focus on individual areas of learning and growth. However, we also want to represent the (multi)conceptual difficulty and (multi)skill capacities relevant to a course.

Example 1. For some AP courses, EKs may be too granular to design an exercise around (like AP GoPo, where LO-based exercises might be the right level), while for others, like AP Macro, some EKs are so rich that we may want 1+ exercises for each.

Example 2. For HS Bio, RNA may be the right granularity for an exercise, with “transcription” and “translation” as the types, because students don’t need to go into those types in real depth in that course.

✅ Do change all of the options in some way in scaled versions.

$\underset{―}{Why?}$‍ So students don’t accidentally see an option and remember it as “right” or “wrong” and not realize this is a unique question where the variables and situation has changed. Change the non-essential names, details, colors, or numbers in an option, e.g. “Her car would stall at 15mph” to “His truck would stall at 25mph."

✅ Do operationalize scaling in a way that is suited for collaboration

$\underset{―}{Why?}$‍ We may move more effectively with specialized models where, for instance, one creator makes the base questions, and another creator, contractor, or other focuses on scaling.

✅ Do keep roughly the same number of items across types.

$\underset{―}{Why?}$‍ So students will see the same distribution across types in attempt 1, 2, 3, etc. Otherwise, if there are four items in every type besides type 6, then students won’t be able to practice the full variety of types in their fourth attempt.

✅ Do give internal names to every type and question, as well as external tags.

$\underset{―}{Why?}$‍ So team members (now and future) besides the original creator can get a sense of the logic behind the types and the variety of questions. It also makes it easier to collaborate internally and with external reviewers if we can reference questions and types that are distinguished by descriptive titles we’ve given them. These strategies will vary slightly for each team, so ensure you’ve synced with your team and manager for the best naming/tagging strategy to enable collaboration in your course.

🚫 Don’t create more than ~30-45 min of exercises in a lesson.

$\underset{―}{Why?}$‍ Because if there are many hours of exercises in a lesson, time-sensitive students may skip around and miss important concepts. If there is a need to create more than ~30-45 min of exercise practice, that is likely a good signal that the lesson should be divided up.

✅ Do prioritize scaling procedural questions.

$\underset{―}{Why?}$‍ Because students should have practice performing the exact same procedure (step 1, step 2, step 3) but in different contexts. This is less at risk of “gaming,” because if a student thinks they’ve “gamed” by learning the procedure and applying it consistently, they’ve actually learned what we want them to!

✅ Do re-use components in other exercises (scale across exercises)

$\underset{―}{Why?}$‍ You may have a great graph, image, or passage in an exercise that wouldn’t scale well in that exercise (because it can only be used in one way), but you could apply it to skills and concepts in 3-5 other exercises!

✅ Do constantly refresh your knowledge of how the product and algorithm uses types to create exercises, quizzes, and tests.

$\underset{―}{Why?}$‍ These mechanics change frequently, and they will be essential for understanding how you curate exercises and what experiences that will create for students. While it’s simpler for product mechanics to change (than to reorganize thousands of exercises), we should be aware of how the product is using our content and be in communication with Product if our content design is not aligned with how it is rendered by the product.