_p_q_r`, plus a combined loss plot:
```
models/Wikidata5M_FSDP_signatures/
├── epoch_loss_by_signature_log.png loss curves of all 10 signatures, log scale
├── DeCaL_torchFSDP_Wikidata5M_p0_q0_r0/
├── DeCaL_torchFSDP_Wikidata5M_p0_q0_r1/
├── DeCaL_torchFSDP_Wikidata5M_p0_q0_r3/
├── DeCaL_torchFSDP_Wikidata5M_p0_q1_r0/
├── DeCaL_torchFSDP_Wikidata5M_p0_q1_r2/
├── DeCaL_torchFSDP_Wikidata5M_p0_q3_r4/
├── DeCaL_torchFSDP_Wikidata5M_p1_q1_r1/
├── DeCaL_torchFSDP_Wikidata5M_p1_q2_r4/
├── DeCaL_torchFSDP_Wikidata5M_p1_q3_r3/
└── DeCaL_torchFSDP_Wikidata5M_p2_q2_r3/
```
Each run directory contains the same core files described in §2.2 (`configuration.json`,
`report.json`, `model.pt`, `entity_to_idx.csv`, `relation_to_idx.csv`, `train_set.npy`,
`memory_map_train_set.npy`, `memory_map_details.json`, `epoch_losses.csv`) plus the
evaluation-specific ones:
| File | What it is |
|---|---|
| `valid_set.npy`, `test_set.npy` | Held-out splits as `int32` `(n, 3)` index arrays, in the same vocabulary space as `train_set.npy`. |
| `er_vocab.p` | Pickled `dict` mapping `(head, relation) → [tails]` over all splits. Used as the filter set for filtered link-prediction ranking (tail direction). |
| `re_vocab.p` | Pickled `dict` mapping `(relation, tail) → [heads]` — the head-direction filter set. |
| `ee_vocab.p` | Pickled `dict` mapping `(head, tail) → [relations]`, used for relation-prediction and for constructing negatives. |
| `eval_report.json` | Filtered test-set link-prediction metrics: `MRR`, `H@1`, `H@3`, `H@10`. |
| `fsdp_shard_checkpoint/` | Present in the Wikidata5M and EN_FR_100K runs (4 ranks there, not 8): `meta.json`, `dense_state.pt`, `entity_shard_rank{0..3}_of_4.pt`, `rank_state_rank{0..3}_of_4.pt` — same semantics as §3.2. |
| `model_dense.pt`, `model_entity_shard_{0..3}.pt` | Intermediate per-shard materialisation outputs kept alongside the merged `model.pt`. |
Evaluation-run hyperparameters are shared across all signatures: `DeCaL`,
`embedding_dim = 256`, `num_epochs = 500`, Adam at `lr = 0.1`, `FixedNegSample` with
`neg_ratio = 2`, `random_seed = 1`, `eval_model = "test"`. Trainer is `torchFSDP`
(bfloat16, CPU optimizer offload) for Wikidata5M and EN_FR_100K, and `torchCPUTrainer`
for the small DBpedia50k runs. `batch_size` is 500,000 except Wikidata5M, which uses
3,800,000.
---
## 5. Using the artifacts
### Download
```bash
BASE=https://files.dice-research.org/datasets/WHALE/IEEE_Big_Data
wget -c $BASE/evaluation/DBpedia50k.zip
wget -c $BASE/evaluation/DBpedia50k_models.zip
# large ones — use -c so an interrupted transfer resumes
wget -c $BASE/dbpedia/dbpedia-snapshot-2022-12.tar.gz
```
### Extract
```bash
tar -xzf dbpedia-snapshot-2022-12.tar.gz # -> dbpedia-snapshot-2022-12/
tar -I 'pigz -d' -xf latest-truthy-clean.tar.gz # -> latest-truthy-clean_100_000/
```
Inference only, skipping the 568 GiB of FSDP shards:
```bash
tar -I 'pigz -d' -xf latest-truthy-clean.tar.gz \
--exclude='*/fsdp_shard_checkpoint/*'
```
### Load a model
Loading these run directories requires `dicee` (dice-embeddings), the framework the
models were trained with — .
```bash
# from PyPI — GPU
pip install dicee
# from PyPI — CPU-only
pip install dicee --extra-index-url https://download.pytorch.org/whl/cpu
# or from source, if you want the exact training code and the export/materialise scripts
git clone https://github.com/dice-group/dice-embeddings
cd dice-embeddings && pip install -e .
```
```python
from dicee import KGE
model = KGE(path="dbpedia-snapshot-2022-12") # any run directory in this release
print(model)
# nearest neighbours / scoring
model.predict_topk(h=["http://dbpedia.org/resource/Berlin"],
r=["http://dbpedia.org/ontology/country"],
topk=10)
```
`KGE` reads `configuration.json` to rebuild the model class and signature, `model.pt`
for the weights, and the two `*_to_idx.csv` files for the vocabularies — those four are
the minimum needed for inference.
> The DBpedia and Wikidata entity tables are 150 GiB and 190 GiB respectively. Loading
> either one whole requires a machine with comparable RAM; for smaller machines, read
> the vectors you need directly out of `model.pt` by index, or use the RDF export.
### Read the triple arrays directly
```python
import json, numpy as np
meta = json.load(open("latest-truthy-clean_100_000/memory_map_details.json"))
triples = np.memmap("latest-truthy-clean_100_000/memory_map_train_set.npy",
dtype=meta["dtype"], mode="r", shape=tuple(meta["shape"]))
print(triples.shape, triples[0]) # (1397639131, 3) [head_idx rel_idx tail_idx]
```
---
## 6. Reproducing the runs
```bash
git clone https://github.com/dice-group/dice-embeddings
cd dice-embeddings && pip install -e .
dicee --path_single_kg dbpedia-snapshot-2022-12.ttl.zst \
--model Keci --p 0 --q 1 --embedding_dim 256 \
--trainer torchFSDP --num_epochs 500 --batch_size 500000 --lr 0.05 \
--scoring_technique NegSample --neg_ratio 2 --random_seed 1
```
The exact, fully resolved arguments for every run are in that run's
`configuration.json`; passing those values reproduces the run bit-for-bit given the same
number of FSDP ranks (`deterministic: true`, `random_seed: 1`).