Repository navigation
KeyError: 'Series UID' in TCIA_PROSTATEx_Prostate_MRI_Anatomy_Model.ipynb #2029
Copy link
Copy link
Description
[2025-09-17T02:57:30.769Z] Running ./model_zoo/TCIA_PROSTATEx_Prostate_MRI_Anatomy_Model.ipynb
[2025-09-17T02:57:30.769Z] Checking PEP8 compliance...
[2025-09-17T02:57:31.356Z] Running notebook...
[2025-09-17T02:57:41.458Z] <frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
[2025-09-17T02:57:42.887Z] <frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()
[2025-09-17T02:57:46.245Z] MONAI version: 1.5.0+20.g401ea4a0
[2025-09-17T02:57:46.245Z] Numpy version: 1.24.4
[2025-09-17T02:57:46.245Z] Pytorch version: 2.5.0a0+e000cf0ad9.nv24.10
[2025-09-17T02:57:46.245Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2025-09-17T02:57:46.245Z] MONAI rev id: 401ea4a0af2b9d475989a78e1bae387875fa2384
[2025-09-17T02:57:46.245Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2025-09-17T02:57:46.245Z]
[2025-09-17T02:57:46.245Z] Optional dependencies:
[2025-09-17T02:57:46.245Z] Pytorch Ignite version: 0.4.11
[2025-09-17T02:57:46.245Z] ITK version: 5.4.4
[2025-09-17T02:57:46.245Z] Nibabel version: 5.3.2
[2025-09-17T02:57:46.245Z] scikit-image version: 0.19.3
[2025-09-17T02:57:46.245Z] scipy version: 1.14.0
[2025-09-17T02:57:46.245Z] Pillow version: 11.3.0
[2025-09-17T02:57:46.245Z] Tensorboard version: 2.16.2
[2025-09-17T02:57:46.245Z] gdown version: 5.2.0
[2025-09-17T02:57:46.245Z] TorchVision version: 0.20.0a0
[2025-09-17T02:57:46.245Z] tqdm version: 4.66.5
[2025-09-17T02:57:46.245Z] lmdb version: 1.7.3
[2025-09-17T02:57:46.245Z] psutil version: 6.0.0
[2025-09-17T02:57:46.245Z] pandas version: 2.2.2
[2025-09-17T02:57:46.245Z] einops version: 0.8.0
[2025-09-17T02:57:46.245Z] transformers version: 4.40.2
[2025-09-17T02:57:46.245Z] mlflow version: 3.3.2
[2025-09-17T02:57:46.245Z] pynrrd version: 1.1.3
[2025-09-17T02:57:46.245Z] clearml version: 2.0.3rc0
[2025-09-17T02:57:46.245Z]
[2025-09-17T02:57:46.245Z] For details about installing the optional dependencies, please visit:
[2025-09-17T02:57:46.245Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2025-09-17T02:57:46.245Z]
[2025-09-17T02:57:48.887Z] papermill --progress-bar --log-output -k python3
[2025-09-17T02:57:49.179Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
[2025-09-17T02:57:49.179Z] warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")
[2025-09-17T02:57:49.179Z]
[2025-09-17T02:57:50.175Z] Executing: 0%| | 0/32 [00:00<?, ?cell/s]
[2025-09-17T02:59:11.808Z] Executing: 3%|▎ | 1/32 [00:00<00:27, 1.11cell/s]
[2025-09-17T02:59:24.154Z] Executing: 22%|██▏ | 7/32 [01:20<04:59, 12.00s/cell]
[2025-09-17T02:59:27.504Z] Executing: 28%|██▊ | 9/32 [01:34<03:59, 10.43s/cell]
[2025-09-17T02:59:28.907Z] Executing: 38%|███▊ | 12/32 [01:38<02:17, 6.87s/cell]
[2025-09-17T02:59:33.159Z] Executing: 44%|████▍ | 14/32 [01:39<01:34, 5.23s/cell]
[2025-09-17T02:59:33.159Z] Executing: 44%|████▍ | 14/32 [01:43<02:12, 7.38s/cell]
[2025-09-17T02:59:33.159Z] /usr/local/lib/python3.10/dist-packages/papermill/iorw.py:149: UserWarning: the file is not specified with any extension : -
[2025-09-17T02:59:33.159Z] warnings.warn(f"the file is not specified with any extension : {os.path.basename(path)}")
[2025-09-17T02:59:33.159Z] Traceback (most recent call last):
[2025-09-17T02:59:33.159Z] File "/usr/local/bin/papermill", line 7, in <module>
[2025-09-17T02:59:33.159Z] sys.exit(papermill())
[2025-09-17T02:59:33.159Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
[2025-09-17T02:59:33.159Z] return self.main(*args, **kwargs)
[2025-09-17T02:59:33.159Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
[2025-09-17T02:59:33.159Z] rv = self.invoke(ctx)
[2025-09-17T02:59:33.159Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
[2025-09-17T02:59:33.159Z] return ctx.invoke(self.callback, **ctx.params)
[2025-09-17T02:59:33.159Z] File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
[2025-09-17T02:59:33.159Z] return __callback(*args, **kwargs)
[2025-09-17T02:59:33.159Z] File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
[2025-09-17T02:59:33.159Z] return f(get_current_context(), *args, **kwargs)
[2025-09-17T02:59:33.159Z] File "/usr/local/lib/python3.10/dist-packages/papermill/cli.py", line 235, in papermill
[2025-09-17T02:59:33.159Z] execute_notebook(
[2025-09-17T02:59:33.159Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 131, in execute_notebook
[2025-09-17T02:59:33.159Z] raise_for_execution_errors(nb, output_path)
[2025-09-17T02:59:33.159Z] File "/usr/local/lib/python3.10/dist-packages/papermill/execute.py", line 251, in raise_for_execution_errors
[2025-09-17T02:59:33.159Z] raise error
[2025-09-17T02:59:33.159Z] papermill.exceptions.PapermillExecutionError:
[2025-09-17T02:59:33.159Z] ---------------------------------------------------------------------------
[2025-09-17T02:59:33.159Z] Exception encountered at "In [5]":
[2025-09-17T02:59:33.159Z] ---------------------------------------------------------------------------
[2025-09-17T02:59:33.159Z] KeyError Traceback (most recent call last)
[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py:3805, in Index.get_loc(self, key)
[2025-09-17T02:59:33.159Z] 3804 try:
[2025-09-17T02:59:33.159Z] -> 3805 return self._engine.get_loc(casted_key)
[2025-09-17T02:59:33.159Z] 3806 except KeyError as err:
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc()
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] File index.pyx:196, in pandas._libs.index.IndexEngine.get_loc()
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] File pandas/_libs/hashtable_class_helper.pxi:7081, in pandas._libs.hashtable.PyObjectHashTable.get_item()
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] File pandas/_libs/hashtable_class_helper.pxi:7089, in pandas._libs.hashtable.PyObjectHashTable.get_item()
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] KeyError: 'Series UID'
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] The above exception was the direct cause of the following exception:
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] KeyError Traceback (most recent call last)
[2025-09-17T02:59:33.159Z] Cell In[5], line 4
[2025-09-17T02:59:33.159Z] 1 dicom_data_dir = "tciaDownload"
[2025-09-17T02:59:33.159Z] 3 # The series_uid defines their directory where the MR data was stored on disk.
[2025-09-17T02:59:33.159Z] ----> 4 mr_series_uid = df.at[df.Modality.eq("MR").idxmax(), "Series UID"]
[2025-09-17T02:59:33.159Z] 5 mr_dir = os.path.join(dicom_data_dir, mr_series_uid)
[2025-09-17T02:59:33.159Z] 7 # Read the DICOM MR series' objects and reconstruct them into a 3D ITK image.
[2025-09-17T02:59:33.159Z] 8 # The itk.F option is added to store the image in memory using floating-point precision pixels (useful if you will filter the image or use it with MONAI).
[2025-09-17T02:59:33.159Z] 9 # For more info on imread, see https://itkpythonpackage.readthedocs.io/en/master/Quick_start_guide.html.
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py:2575, in _AtIndexer.__getitem__(self, key)
[2025-09-17T02:59:33.159Z] 2572 raise ValueError("Invalid call for scalar access (getting)!")
[2025-09-17T02:59:33.159Z] 2573 return self.obj.loc[key]
[2025-09-17T02:59:33.159Z] -> 2575 return super().__getitem__(key)
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/indexing.py:2527, in _ScalarAccessIndexer.__getitem__(self, key)
[2025-09-17T02:59:33.159Z] 2524 raise ValueError("Invalid call for scalar access (getting)!")
[2025-09-17T02:59:33.159Z] 2526 key = self._convert_key(key)
[2025-09-17T02:59:33.159Z] -> 2527 return self.obj._get_value(*key, takeable=self._takeable)
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/frame.py:4214, in DataFrame._get_value(self, index, col, takeable)
[2025-09-17T02:59:33.159Z] 4211 series = self._ixs(col, axis=1)
[2025-09-17T02:59:33.159Z] 4212 return series._values[index]
[2025-09-17T02:59:33.159Z] -> 4214 series = self._get_item_cache(col)
[2025-09-17T02:59:33.159Z] 4215 engine = self.index._engine
[2025-09-17T02:59:33.159Z] 4217 if not isinstance(self.index, MultiIndex):
[2025-09-17T02:59:33.159Z] 4218 # CategoricalIndex: Trying to use the engine fastpath may give incorrect
[2025-09-17T02:59:33.159Z] 4219 # results if our categories are integers that dont match our codes
[2025-09-17T02:59:33.159Z] 4220 # IntervalIndex: IntervalTree has no get_loc
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/frame.py:4638, in DataFrame._get_item_cache(self, item)
[2025-09-17T02:59:33.159Z] 4633 res = cache.get(item)
[2025-09-17T02:59:33.159Z] 4634 if res is None:
[2025-09-17T02:59:33.159Z] 4635 # All places that call _get_item_cache have unique columns,
[2025-09-17T02:59:33.159Z] 4636 # pending resolution of GH#33047
[2025-09-17T02:59:33.159Z] -> 4638 loc = self.columns.get_loc(item)
[2025-09-17T02:59:33.159Z] 4639 res = self._ixs(loc, axis=1)
[2025-09-17T02:59:33.159Z] 4641 cache[item] = res
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] File /usr/local/lib/python3.10/dist-packages/pandas/core/indexes/base.py:3812, in Index.get_loc(self, key)
[2025-09-17T02:59:33.159Z] 3807 if isinstance(casted_key, slice) or (
[2025-09-17T02:59:33.159Z] 3808 isinstance(casted_key, abc.Iterable)
[2025-09-17T02:59:33.159Z] 3809 and any(isinstance(x, slice) for x in casted_key)
[2025-09-17T02:59:33.159Z] 3810 ):
[2025-09-17T02:59:33.159Z] 3811 raise InvalidIndexError(key)
[2025-09-17T02:59:33.159Z] -> 3812 raise KeyError(key) from err
[2025-09-17T02:59:33.159Z] 3813 except TypeError:
[2025-09-17T02:59:33.159Z] 3814 # If we have a listlike key, _check_indexing_error will raise
[2025-09-17T02:59:33.159Z] 3815 # InvalidIndexError. Otherwise we fall through and re-raise
[2025-09-17T02:59:33.159Z] 3816 # the TypeError.
[2025-09-17T02:59:33.159Z] 3817 self._check_indexing_error(key)
[2025-09-17T02:59:33.159Z]
[2025-09-17T02:59:33.159Z] KeyError: 'Series UID'
Reactions are currently unavailable
Metadata
Metadata
Labels
No labels
Type
Projects
- StatusShow more project fieldsDone
Activity
ericspod commentedon Sep 18, 2025
I'm looking into this, but one issue I'm seeing is that
pydicom-seghasn't been updated in 2 years. This might be a dead project we shouldn't rely on as a dependency.ericspod commentedon Sep 18, 2025
The table appears to be presenting the Series UID in the column "Series ID" now, I think changing the column selection in various places to that string will work.
ericspod commentedon Sep 18, 2025
We also need to add
huggingface_hubas a new dependency to install, anditkwidgetsmight require a different version of zarr versus other dependencies and causing a conflict in some configurations.Skip tcia notebook test for 1.5.1 release (#2033)
kirbyju commentedon Sep 22, 2025
I can't figure out how to fix the issues related to huggingface_hub or itkwidgets. Maybe @thewtex or @aylward can help with itkwidgets as I didn't write that part of the notebook. I think I can resolve the parts related to tcia_utils (changing to "Series ID" works) and get rid of the dependency on pydicom_seg.
thewtex commentedon Sep 23, 2025
itkwidgets does require zarr-2.
For reading DICOM SEG, another option is
itkwasm-dicom:https://pypi.org/project/itkwasm-dicom/
https://itk-wasm-dicom-python-docs.on.fleek.co/apidocs/itkwasm_dicom/itkwasm_dicom.read_segmentation.html
kirbyju commentedon Sep 23, 2025
I think the changes shown in https://colab.research.google.com/drive/1N7dFgTKqVAXmGtL2RXlvd05WkC648zBP?usp=sharing address the data download bug, eliminate the need for pydicom_seg (resulting nifti files seem to open fine in Slicer), and clean up other unnecessary complexity in the installations/imports. I'm still stuck on itkwidgets though. Nothing happens when I run that cell even after upgrading to v55. Also, it seems like the huggingface part may be more of a "nice to have" thing because as far as I could tell the model still downloads and runs properly?
thewtex commentedon Sep 23, 2025
For the itkwidgets install line, does adding
zarr<3help?kirbyju commentedon Sep 23, 2025
@ericspod I just tried on Jupyter instead of Colab and ran into the huggingface issue. Didn't realize that was pre-installed by Colab.
@thewtex adding zarr<3 seemed like it would help. At least in Colab it displayed the plugin icon, but it hung there indefinitely and never loaded the images/seg:
But in Jupyter I'm just getting this:
ericspod commentedon Sep 26, 2025
Hi @kirbyju and @thewtex thanks for the help on this one. This notebook could be amended to force zarr-2 installation but the itkwidgets problem is possibly still there. I've copied over from the example Colab link any changes with slight updates. On Windows I ran everything (just about) with Jupyter (not Jupyterlab or through vscode) after running the install cell, installing the plugin, and restarting the notebook. The following raises an exception for me:
This raises
TypeError: in method 'itkResampleImageFilterIF3IF3_SetInterpolator', argument 2 of type 'itkInterpolateImageFunctionIF3D *', meaning it doesn't like theinterpolatorobject. If I omit theInterpolatorargument this works but will produce differing results I assume. Following cells run fine showing the itk viewers as expected. We still have a challenge that this doesn't render in Colab but does technically work.I also noticed that I do require Huggingface with MONAI 1.5.1, we don't correctly handle the fallback case when huggingface_hub is not installed so this will need addressing in the next release. I got it to work by just installing
huggingface_hubwhich ended up working correctly for me.My full Python 3.12.11 environment
ericspod commentedon Sep 26, 2025
I opened a PR on this issue if you could review, thanks.
thewtex commentedon Sep 26, 2025
@ericspod awesome work! LGTM.
We may be able to resolve the type issue with:
or
(untested)
ericspod commentedon Oct 29, 2025
Hi @thewtex sorry for the delay, I have this fix integrated into #2037 now. Thanks!
Updating TCIA MRI Anatomy Model Notebook (#2037)
kirbyju commentedon Oct 30, 2025
Hi all, I sincerely apologize for the back and forth on this Series UID thing, but after this issue was discovered the underlying NBIA API was recently changed in v4 to fix the inconsistencies in the fields returned and the newest release of tcia_utils was updated to match. It is now "SeriesInstanceUID" across the board for all queries that return series-level DICOM metadata. Unfortunately, we didn't have tcia_utils pinned to any particular version in the notebook so that update is breaking the notebook again due to using "Series UID" in 2 cells. I don't anticipate any more issues with this kind of thing moving forward, but it might still make sense to pin to the v3.2.1 of tcia_utils just in case?
I also tested Eric's other changes just now. In Colab it renders the imjoy plugin logo as if it's going to work, but then never loads the actual image data for me. Despite that, it proceeds to the inference step and then hangs indefinitely (or at least for 13min so far). I'm using a free Colab account in case that matters. It doesn't seem to be running out of ram. I have it set to a t4 GPU runtime.
On my local macbook Jupyter installation I'm having very similar problems (imjoy not working, hanging on inference):
ericspod commentedon Oct 30, 2025
Thanks for looking into this, is there a new fix we should do a PR for? If you wanted to propose that we can merge that before it causes CI issues.
As for the plugin, it might be sensitive to what browser you use, platform, and other factors to determine if the plugin works. I managed to get things running as you see in the notebook updates using Firefox in Windows, perhaps trying a different browser or OS would be an idea if you can. You can also look at the Javascript console in your browser's developer panel to see if anything is printed there to indicate why the plugin fails.
kirbyju commentedon Oct 31, 2025
I submitted the suggested fixes related to tcia_utils.
Looking again at the itkwidgets issue, I'm still not having any luck with itkwidgets on Firefox or Chrome for Colab on my Macbook, but I got it working in my Jupyter Lab local system.
ericspod commentedon Nov 4, 2025
I hope this isn't a problem for colab in general but we'll have to do some more testing to see where the problem is coming from with the interaction of platforms and browsers. I see the PR for the notebook fix, thanks.
Fix tcia_utils issues loading metadata df to support latest v3.2.1 re…