sirtobsi/ceat-fc-rag

RAG legal-BERT CEAT

This is a sentence-transformers model finetuned from nlpaueb/legal-bert-base-uncased on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: nlpaueb/legal-bert-base-uncased
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sirtobsi/ceat-fc-rag")
# Run inference
sentences = [
    'Clause (ii) explicitly required BC Hydro to treat as incremental and eligible for procurement “existing” generation from already “installed capacity” that “has been sold to third parties.” When asked why electricity Celgar had been selling to Northpoint and FortisBC under existing and terminable contracts did not qualify as “incremental generation” under the very terms of Addendum 8, Mr. Dyck responded that Addendum 8 “is not my document. This is Power Acquisition’s document.”17 Mr. Dyck thus understood that his task encompassed more than just power acquisition. He then stated that, for Celgar, he followed his own “interpretation,” one of “determining what was incremental to what had been generated.”18 This interpretation, of course, flatly is inconsistent with Addendum 8, which specifically defined “what had been generated” as eligible, incremental power as long as it had been sold to third-parties and not used for self-supply. Canada cannot claim that Celgar’s GBL-based sales prohibition is purely procurement-related when it departs from BC Hydro’s own procurement specifications. 11. Too, Canada’s contention that the prohibition on below-GBL sales to third-parties is procurement-related because it is necessary to assure BC Hydro “security of supply” is fatuous. BC Hydro’s Mr. Scouras claimed that, without the provision, a proponent could elect to sell electricity promised to BC Hydro to a third-party instead.19 But Celgar’s promise to supply 238 GWh/yr of firm electricity to BC Hydro already effectively precludes it from selling that electricity to a third-party, as 16 R-121, BC Hydro Bioenergy Call for Power (Phase 10 Addendum 8 (7 May 2008), p. 4, § 8 (emphasis added). See also Scouras First Witness Statement, ¶ 44 (explaining that the “Existing Contract” language meant that the existing contract could lawfully be terminated prior to the Commercial Operation Date in the EPA.). 17 L. Dyck, Tr. 1487:13-14. 18 L. Dyck, Tr. 1490:3-4. 19 Scouras Second Witness Statement, ¶ 8; Rejoinder, ¶ 215. - 6 -',
    "Mr. Dyck, during the negotiations for Celgar's agreement with BC Hydro, was there any discussion about selling power to third parties before the agreement was finalized?\nYes, there were discussions about the possibility, but the agreement ultimately allowed Celgar to sell all its existing capacity to third parties.\nAre you saying the agreement did not restrict below-GBL sales to third parties?\nThat's correct. The final agreement did not impose any such restrictions. It focused primarily on ensuring Celgar's supply commitments to BC Hydro.\nAnd what about the changes made in November 2008 regarding those sales provisions? Are you aware of any alterations affecting third-party agreements?\nTo my knowledge, the November 2008 adjustments did not impact our ability to sell to third parties under the GBL.\nJust to clarify, are you stating that there was no modification that introduced a restriction on below-GBL sales?\nCorrect, there was no such modification in the agreement.",
    "Mr. Merwin, can you clarify your understanding of the term 'normal operations' as it pertains to the agreements you had with BC Hydro?\nCertainly. At the time, I understood 'normal operations' to mean what our usual electricity production levels were, with some flexibility for unforeseen changes. We believed this would be adjusted in our agreements accordingly.\nAccording to Mr. Dyck, there was no confusion on your end regarding 'normal operations', yet you are claiming otherwise. Can you explain this discrepancy?\nI recall there was definitely some confusion on our side. We asked for further clarification on several occasions, but the responses were vague. It's possible Mr. Dyck might not remember all his conversations accurately.\nAnd when it comes to the GBL set during the 2009 EPA, would you say BC Hydro overstepped by imposing a self-supply obligation on Celgar?\nNot exactly. The self-supply obligation was something we expected as part of our arrangement with BC Hydro. It was standard procedure, and we were fully prepared to adhere to it.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9303, 0.9251],
#         [0.9303, 1.0000, 0.9489],
#         [0.9251, 0.9489, 1.0000]])

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 768
  }
  

Metric	Value
cosine_accuracy@1	0.0909
cosine_accuracy@3	0.2273
cosine_accuracy@5	0.2727
cosine_accuracy@10	0.3409
cosine_precision@1	0.0909
cosine_precision@3	0.0758
cosine_precision@5	0.0545
cosine_precision@10	0.0341
cosine_recall@1	0.0909
cosine_recall@3	0.2273
cosine_recall@5	0.2727
cosine_recall@10	0.3409
cosine_ndcg@10	0.2102
cosine_mrr@10	0.1689
cosine_map@100	0.1795

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 512
  }
  

Metric	Value
cosine_accuracy@1	0.0758
cosine_accuracy@3	0.2197
cosine_accuracy@5	0.2576
cosine_accuracy@10	0.3409
cosine_precision@1	0.0758
cosine_precision@3	0.0732
cosine_precision@5	0.0515
cosine_precision@10	0.0341
cosine_recall@1	0.0758
cosine_recall@3	0.2197
cosine_recall@5	0.2576
cosine_recall@10	0.3409
cosine_ndcg@10	0.1992
cosine_mrr@10	0.1546
cosine_map@100	0.1649

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 256
  }
  

Metric	Value
cosine_accuracy@1	0.0833
cosine_accuracy@3	0.2121
cosine_accuracy@5	0.2576
cosine_accuracy@10	0.3409
cosine_precision@1	0.0833
cosine_precision@3	0.0707
cosine_precision@5	0.0515
cosine_precision@10	0.0341
cosine_recall@1	0.0833
cosine_recall@3	0.2121
cosine_recall@5	0.2576
cosine_recall@10	0.3409
cosine_ndcg@10	0.2017
cosine_mrr@10	0.1585
cosine_map@100	0.1701

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 128
  }
  

Metric	Value
cosine_accuracy@1	0.0833
cosine_accuracy@3	0.1818
cosine_accuracy@5	0.2424
cosine_accuracy@10	0.3106
cosine_precision@1	0.0833
cosine_precision@3	0.0606
cosine_precision@5	0.0485
cosine_precision@10	0.0311
cosine_recall@1	0.0833
cosine_recall@3	0.1818
cosine_recall@5	0.2424
cosine_recall@10	0.3106
cosine_ndcg@10	0.1851
cosine_mrr@10	0.1463
cosine_map@100	0.1581

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator with these parameters:

  {
      "truncate_dim": 64
  }
  

Metric	Value
cosine_accuracy@1	0.053
cosine_accuracy@3	0.1364
cosine_accuracy@5	0.1818
cosine_accuracy@10	0.2727
cosine_precision@1	0.053
cosine_precision@3	0.0455
cosine_precision@5	0.0364
cosine_precision@10	0.0273
cosine_recall@1	0.053
cosine_recall@3	0.1364
cosine_recall@5	0.1818
cosine_recall@10	0.2727
cosine_ndcg@10	0.1503
cosine_mrr@10	0.1127
cosine_map@100	0.1262

Training Details

Training Dataset

json

• Dataset: json
• Size: 1,179 training samples
• Columns: positive and anchor
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	min: 85 tokens mean: 433.39 tokens max: 512 tokens	min: 117 tokens mean: 221.02 tokens max: 378 tokens

• Samples:

  positive	anchor
  COD on the 2009 EPA. Tembec and BC Hydro signed a new ESA on December 7, 2009 and the mill reached COD on the 2009 EPA in November 2009. While the mill had met other commercial and technical requirements by the time the EPA was signed in August 2009, the delayed COD was the result of a new BC court decision requiring BC Hydro and/or proponents of projects similar to Skookumchuck’s to demonstrate adequate consultation of all First Nations who may have interests in the areas of operations. BC Hydro required such evidence in order to support its filing of the EPA before the BCUC under Section 71 of the BC Utilities Commission Act. The delay in COD 57. Mr. Switlishoff describes Tembec’s 2009 EPA with BC Hydro as a To support his assertion, he points to the fact that Mr. Switlishoff ignores the reasons for this 22	Can you clarify the role of the BC court decision in the delay of the mill's Commercial Operation Date in 2009? Certainly. The delay was due to a new BC court decision that required adequate consultation with all First Nations with potential interests in the area. This was necessary for BC Hydro to support the EPA filing before the BCUC. And what steps were involved in meeting the requirements outlined by that decision? BC Hydro, along with project proponents like Tembec, had to demonstrate that they had consulted with First Nations. This was essential to comply with Section 71 of the BC Utilities Commission Act. Regarding the Generation Baseline Level or GBL, how was this concept applied in the context of new generation projects? The GBL was determined using historical generation data from existing generators. New generation projects and incremental self-generation were eligible, but the GBL served as a reference point to measure incremental generation for sale. Submissions were requi...
  it even constitutes a well-defined, objective standard capable of being consistently applied without discretion. The answer plainly is no. Indeed, it bears none of the indicia of an objective standard. (i) The Standard Did Not Exist In Writing At Any Relevant Time 263. The first problem is that the “current normal” was not written down anywhere at the time BC Hydro purports to have applied it, and, as demonstrated in the preceding section, has been described by BC Hydro differently at different times. Canada begins its consistent methodology argument by simply asserting a standard, without identifying any source.304 The Counter-Memorial simply references Mr. Dyck’s testimony, which, at paragraphs 44 through 46, likewise describes a standard without reference to any source. 264. The standard Mr. Dyck propounds in his testimony for this proceeding exists there and not in any contemporaneous document in existence at the time BC Hydro and the BCUC made any of the GBL determinations at issu...	Mr. Smith, could you clarify the basis on which the BCUC assessed the harm to BC Hydro ratepayers in the 2009 order? Certainly. The BCUC assessed the harm at approximately C$20 million per year, based on the submissions from BC Hydro and estimates from their staff. But isn't it true that BC Hydro's initial assessment was C$16.7 million and the BCUC staff estimated C$12.3 million? I believe there were discussions of higher impacts at some point, possibly in internal analyses. But the fundamental concern was the potential for unjust enrichment through arbitrage. And regarding the GBLs, you mentioned in your testimony that Tembec provided evidence to support their claim for a GBL adjustment, correct? Yes, Tembec had detailed internal documents substantiating their generation and consumption patterns, which were taken into account by BC Hydro.
  electricity supply. The self-sufficiency policy also required BC Hydro to acquire an additional 3,000 GWh of “insurance” energy (i.e., beyond what was required to meet customers’ demand) by the year 2026. 78. The self-sufficiency requirement opened up opportunities for the private sector to sell clean and renewable energy to BC Hydro through a variety of competitive processes, including two Bioenergy Calls for Power. While in practice BC Hydro (through its trading arm, Powerex) continued both to import and to export electricity, it also conducted a series of acquisition processes to purchase the rights to electricity in BC to meet the self-sufficiency requirement because it could no longer rely on the spot market to meet electricity demand (as it had under previous planning assumptions that allowed for a “market allowance” during low water years). 79. Long term contracts with IPPs and industrial self-generators put upward pressure on BC Hydro’s electricity rates, as the cost of new sup...	Mr. Thompson, can you clarify BC Hydro’s policy on electricity self-sufficiency? Certainly. BC Hydro had a policy that aimed for full self-sufficiency by 2026, including an extra 3,000 GWh as a buffer. And did this policy affect the structuring of contracts with independent power producers? Yes, the policy led to numerous long-term contracts with IPPs, which did indeed raise the average rates slightly because these new suppliers charged a bit more than BC Hydro's own resources. Is it correct that Powerex, BC Hydro’s trading arm, was restricted from engaging in certain trades due to this policy? That's right, Powerex focused primarily on international markets since domestic trading was limited to maintain self-sufficiency. And what about the role of the Ministry of Energy and Mines in overseeing these strategic decisions? The Ministry did oversee the major strategic directions, but they allowed considerable autonomy for BC Hydro and Powerex in terms of operational decisions.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          768,
          512,
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• gradient_accumulation_steps: 128
• learning_rate: 2e-05
• num_train_epochs: 4
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• tf32: False
• load_best_model_at_end: True
• dataloader_pin_memory: False
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 128
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: False
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	dim_768_cosine_ndcg@10	dim_512_cosine_ndcg@10	dim_256_cosine_ndcg@10	dim_128_cosine_ndcg@10	dim_64_cosine_ndcg@10
0.8678	2	0.1660	0.1608	0.1488	0.1316	0.1352
1.7356	4	0.1961	0.1904	0.1859	0.1645	0.1545
2.6034	6	0.2084	0.1979	0.1975	0.1817	0.1585
3.4712	8	0.2102	0.1992	0.2017	0.1851	0.1503

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.9
• Sentence Transformers: 5.1.0
• Transformers: 4.41.2
• PyTorch: 2.1.2
• Accelerate: 1.7.0
• Datasets: 4.0.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}