docs/WIP/Tutorial_DiscordMERLIN.ipynb is outdated

[NimaSarajpoor]

The notebook Tutorial_DiscordMERLIN.ipynb is outdated. For instance:

• It loads data from directories that do not exist. Also, the original data provided by authors in the MERLIN webpage is not accessible anymore.
• It uses some code from STUMPY that have been updated since then, and they are not backward compatible. For example, the notebook uses the code T, M_T, Σ_T = core.preprocess(ts, m); However, this function in branch main now returns four values instead of three.

[ShreyasK06]

Hi! I'm a sophomore at Penn State studying CS, currently doing independent research on time series data from sensor signals, so this is right up my alley. I'm also a student in the CodePath AI301 course, looking to make open source contributions.

I'd like to take this on! My plan:

1. Find a publicly accessible replacement dataset that demonstrates discord/anomaly detection (likely from UCI or a synthetic example)
2. Fix the core.preprocess call to unpack all four return values correctly
3. Run the notebook end-to-end and fix any other breaking changes
4. Submit a PR with the updated notebook

Could I be assigned? Happy to share my progress as I go.

[NimaSarajpoor]

@ShreyasK06
Welcome to STUMPY's community! And, thanks for sharing your interest in addressing this issue!!

My plan: ...

Could I be assigned? Happy to share my progress as I go

The plan sounds good! You are more than welcome to work on this and submit a PR!

Find a publicly accessible replacement dataset that demonstrates discord/anomaly detection (likely from UCI or a synthetic example)

If locating a publicly-available replacement is difficult, we can try to recreate the synthetic noisy-sine example shown in the first figure of the notebook. Note that this plot is shown in the MERLIN paper (Fig. 4)

[ShreyasK06]

Hi, I've done some more investigation into the broken core.preprocess calls and wanted to share an updated picture before moving forward.

After looking at the current source more carefully, the situation is slightly more involved than the issue description suggests. The issue notes that core.preprocess now returns four values instead of three, but the current main branch actually returns six values: T, M_T, Σ_T_inverse, M_T_m_1, T_subseq_isfinite, and T_subseq_isconstant. So the notebook is further out of date than originally reported, which makes the solving it harder, I have come up with two approaches.

Option A — Update the internal core.preprocess calls directly

The first one is updating the unpacking line from the old 3-value form to handle all six return values. However, this is risky because the old notebook used Σ_T as a plain standard deviation, but the equivalent slot in the new return signature is Σ_T_inverse. That means any downstream math relying on Σ_T would produce wrong results even after fixing the unpack, so this it would require carefully checking every variable derived from core.preprocess through the rest of the notebook.

Option B — Replace internal calls with the public API

Since core.preprocess is an internal function not intended for end users, I could rewrite the affected cells using the appropriate public-facing STUMPY functions like stumpy.stump. This would make the notebook more resilient to future internal changes, and more useful as a tutorial since it shows the API users are actually meant to call.

I have also found a dataset that I think would be a good fit, I'd like to use scipy.datasets.electrocardiogram, there is no download r
required, It's built directly into SciPy, which is already a STUMPY dependency.

It has real anomalies with known locations,The signal is a 5-minute ECG recording (108,000 data points at 360 Hz) from the MIT-BIH Arrhythmia Database via PhysioNet. It explicitly contains noise artifacts, normal heartbeats, and pathological changes, meaning there are discords already in the data that MERLIN can find and that are visually interpretable in a tutorial context. It also has a permanent academic reference (MIT-BIH / PhysioNet), so it won't disappear the way the original MERLIN dataset page did.

Let me know if this direction sounds good and I'll get started on a PR!

[NimaSarajpoor]

@ShreyasK06

The issue notes that core.preprocess now returns four values instead of three, but the current main branch actually returns six values: T, M_T, Σ_T_inverse, M_T_m_1, T_subseq_isfinite, and T_subseq_isconstant.

The function core.preprocess returns 4 values. Maybe we are talking about two different things here. Can you please provide the permalink of the code snippet you are referring to?

[NimaSarajpoor]

@ShreyasK06
Also, this is generally how a process looks in STUMPY:

T (and other inputs) → preprocessing → processed T (and other derived values) → analysis → ...

One of STUMPY's design principles (thanks to the owner, @seanlaw) is that users can compose different building blocks to create new workflows. MERLIN is a new workflow that uses the core.preprocess building block. In contrast, stumpy.stump is a complete workflow that happens to use its own set of underlying building blocks.

Because of this, we likely cannot simply replace core.preprocess with stumpy.stump (or another public API). core.preprocess performs only the preprocessing step, whereas stumpy.stump represents a larger end-to-end process.
Instead, we should determine how the newly added return value from core.preprocess affects the remainder of the MERLIN workflow and update the implementation accordingly.

[ShreyasK06]

My apologies for the confusion in my earlier comments. I was mistakenly looking at core.preprocess_diagonal instead of core.preprocess, which led me to incorrectly mention 6 return values. I also undertand that replacing core.preprocess with stumpy.stump is wrong and I understand that core.preprocess is an intentional building block in the MERLIN workflow and that swapping it out for an end-to-end function would break the design. I will update every occurrence of this in the notebook to the 4-value form, and then trace how T_subseq_isconstant needs to be threaded through the rest of the MERLIN workflow, in particular, anywhere the notebook performs distance calculations,to make sure it is handled correctly end to end.

[NimaSarajpoor]

@ShreyasK06

I will update every occurrence of this in the notebook to the 4-value form, and then trace how T_subseq_isconstant needs to be threaded through the rest of the MERLIN workflow, in particular, anywhere the notebook performs distance calculations

Sounds good! And to understand how this can affect the z-normalized distance between two subsequences, see core._calculate_squared_distance. In nutshell:

• If two subsequences are both constant --> their z-normalized euclidean distance is 0
• If only one of the two subsequences is constant --> their z-normalized euclidean distance is np.sqrt(m), where m is the window size.

[ShreyasK06]

Quick update on where things stand, plus a few things I want to flag before opening the PR.
Fix 1 — core.preprocess

This part's done. I updated both T, M_T, Σ_T = core.preprocess(...) calls to the correct 4-value unpack (T, M_T, Σ_T, T_subseq_isconstant = core.preprocess(...)), and threaded T_subseq_isconstant through _find_candidates, _get_approx_P, and _refine_candidates, updating the call sites so it gets passed correctly into core._mass and _prescrump.

Dataset — switched away from scipy.datasets.electrocardiogram

I'd planned to use scipy.datasets.electrocardiogram(), but it needs pooch as an optional dependency, and it just failed outright:
ImportError: Missing optional dependency 'pooch' required for scipy.datasets module.
pooch isn't in STUMPY's dependencies anywhere (requirements.txt, pyproject.toml), and every other WIP notebook in the repo pulls data from URLs instead of installing extra packages. Adding a new dependency just for this notebook didn't seem right, so I went with URL-based loading instead. There's no working URL left for the original NoisySine dataset, so I replaced it with a synthetic signal built in NumPy — a sinusoid with noise and a spike anomaly — which does the same job for the tutorial without adding anything new to install.

A few more things I found while testing

NYC taxi loading is broken too. It points to a local path that doesn't exist anymore. I'd swap it for a direct pd.read_csv against the NAB GitHub repo (https://raw.githubusercontent.com/numenta/NAB/master/data/realKnownCause/nyc_taxi.csv) — same dataset, just pulled from its actual public source.

core._sliding_dot_product got renamed. Commit bd6aed8 moved it out of core into a new stumpy/sdp module and dropped the underscore, so it's now core.sliding_dot_product. The notebook still calls the old name and throws AttributeError. Two call sites need updating — _find_candidates and _refine_candidates.

np.NINF doesn't exist anymore. NumPy 2.0 removed it, and the notebook uses it as a sentinel in a few places — _refine_candidates, the MP verification block, _discords, stumpy_top_k_discords. On any current NumPy install these cells just break. It's a clean swap, np.NINF → -np.inf, 8 spots across 4 cells, no logic changes involved.

Let me know if these three are fine to fold into the same PR, or if you'd rather see them split out on their own.

[seanlaw]

@ShreyasK06 For the taxi cab dataset, please use:

df_taxi = pd.read_csv("https://zenodo.org/record/4276428/files/STUMPY_Basics_Taxi.csv?download=1")

[NimaSarajpoor]

@ShreyasK06

Thanks for going through the notebook and sharing the details.

Let me know if these three are fine to fold into the same PR, or if you'd rather see them split out on their own.

Let's address them in the same PR. It’s better to submit changes in small chunks so reviewers can provide more focused and effective feedback.

[ShreyasK06]

While running the notebook end to end to confirm the fixes, I hit one more pre-existing breakage unrelated to the changes already made:

ValueError: could not broadcast input array from shape (1489,1489) into shape (1489,)
This happens when _discords calls _get_approx_P, which internally calls stumpy.scrump._prescrump. It looks like _prescrump now accepts a k parameter (default k=1) and returns a 2D matrix profile of shape (l, k) instead of the 1D (l,) shape the notebook expects. When _get_approx_P returns this 2D array and _discords does include & (approx_P >= r_updated), NumPy broadcasts them into a (1489, 1489) array, which then can't fit into is_cands in _find_candidates.

This is a _prescrump signature change beyond the ones already listed in the PR — the notebook needs to slice or squeeze the output of _prescrump down to 1D before passing it downstream. Before I apply a fix I wanted to check in — is slicing the first column (approx_P[:, 0]) the right approach here, or would you prefer the notebook call _prescrump differently?

[NimaSarajpoor]

@ShreyasK06
Thanks for the effort and working on the fixes. While I can try to remember the code or just put it in my head, I think it is better to start with a PR. You can apply changes there in small chunks and we can review them together. IMO, the goal of having discussion here is not about having the full solution before submitting a PR. It is about having some kind of plan rather than just jumping into solution.

At this point, I believe it is better to submit a PR and push changes in small chunks. Once we see the changes, we can have a better and more productive discussion.