1 Multi-regional Input-Output (MIOM)

The headline feature of version 1.0.0 is the introduction of Multi-regional Input-Output (MIOM) models. This allows analysts to go beyond single economies and study the interdependencies between different nations or regions.

The new miom() class creates a multi-regional model that inherits all the performance and functionality of the single-region iom() class, adding specialized tools for regional analysis:

• Bilateral Trade: Easily extract trade flows between specific regions.
• Regional Multipliers: Compute intra-regional, inter-regional, and spillover multipliers.
• Interdependence Measures: Quantify how much regions rely on each other.

Here is a quick look at how you can analyze global trade interdependencies using the world_2000 dataset (now part of the {fiodata} package):

library(fio)

# Load the multi-regional world dataset
miom_world <- fiodata::world_2000

# Get bilateral trade from Brazil to China
miom_world$get_bilateral_trade("BRA", "CHN")[1:5, 1:2]
#>                                                BRA_Agriculture, Hunting, Forestry and Fishing
#> CHN_Agriculture, Hunting, Forestry and Fishing                                    0.868486382
#> CHN_Mining and Quarrying                                                          0.677175460
#> CHN_Food, Beverages and Tobacco                                                   0.784219678
#> CHN_Textiles, leather and footwear                                                0.137408834
#> CHN_Pulp, paper, printing and publishing                                          0.003052195
#>                                                BRA_Mining and Quarrying
#> CHN_Agriculture, Hunting, Forestry and Fishing              0.001110208
#> CHN_Mining and Quarrying                                    1.703217354
#> CHN_Food, Beverages and Tobacco                             0.005707213
#> CHN_Textiles, leather and footwear                          0.129627291
#> CHN_Pulp, paper, printing and publishing                    0.015560710

# Compute regional interdependence
miom_world$get_regional_interdependence() |> head()
#>   country self_reliance total_spillover_out total_spillover_in
#> 1     AUS      1.968515           0.3168527        0.006511634
#> 2     AUT      1.614535           0.4900724        0.004048698
#> 3     BEL      1.649908           0.7652207        0.011148228
#> 4     BRA      1.918948           0.2328115        0.004065824
#> 5     CAN      1.650380           0.4280919        0.007634451
#> 6     CHN      2.342241           0.2867934        0.016735715
#>   interdependence_index
#> 1             0.1609602
#> 2             0.3035377
#> 3             0.4637960
#> 4             0.1213225
#> 5             0.2593898
#> 6             0.1224440

For a complete overview on miom class and its methods, check out the vignette: Multi-regional Input-Output Analysis.

2 Closing the Model

Input-output models are traditionally “open,” treating households and government as external to the production system. Version 1.0.0 introduces the close_model() method, which allows you to “internalize” these sectors.

By moving households or the government from final demand into the intermediate transactions matrix, you can better capture the “induced” effects of economic shocks—for instance, how an increase in production leads to higher wages, which in turn fuels more consumption.

iom_br <- fiodata::br_2020

# Close the model for household consumption
iom_br$close_model("household")

# Technical coefficients and multipliers now include the induced effects
iom_br$compute_tech_coeff()
iom_br$compute_leontief_inverse()
iom_br$compute_multiplier_output()

3 Direct Data Access

To make it even easier to start your analysis, we’ve added download_wiod(). This function allows you to download World Input-Output Database (WIOD) tables directly from their official source and load them into R. Combined with {fio}’s processing speed, this significantly reduces the time from data collection to insight.

For a complete example on working with WIOD data, check out the vignette: Working with WIOD.

4 The {fiodata} Split

As the package grew, the built-in datasets reached a size that exceeded CRAN’s recommendations. To keep {fio} lightweight and fast to install, we have moved the datasets (br_2020 and world_2000) to a separate companion package: {fiodata}.

When you install {fio}, you might want to install {fiodata} as well to follow along with our examples:

# Install from CRAN
install.packages("fiodata")

5 Acknowledgements

This release would not have been possible without the robust ecosystem of R and Rust packages. Special thanks to the developers of the extendr and faer projects, which provide the foundation for {fio}’s hybrid architecture.

For a full list of changes, please check the NEWS.md file. Happy analyzing!

1 ECONOMICS AS AN EXACT SCIENCE

Still in their introduction, the authors argue about the position of economic science as an exact or human science:

The first question that arises in the mind of the layman is whether economics is an exact science or a science in the field of the so-called humanities (human sciences). If we had to choose between one framework or another, we would say that economics belongs to the field of human sciences. However, it occupies a special position among the human sciences: a part of the scientific contributions of academic economists uses the method of exact sciences. Thus, it can be safely stated that there is an exact field within the scope of investigations made by economists. Not all studies offered by economists, however, appear as an effort in exact science. But an important part of them, indeed, the most expressive, employs the rigorous mathematical methods typical of the exact natural sciences that have physics as their central model.

These exact methods would have social norms as their object, reinforced by penal laws, which “encompass all types of regular and standardized behavior, equal for all individuals within a group.” However, the authors themselves recognize the limited scope of economic science in the study of these social norms:

For the economist, a few types of institutions, seen as stylized types, are sufficient, such as private property (which induces agents to respect property rights and contracts) and the optimizing agent (who maximizes utility or profit), or else communism (which coerces them to accept collective property) and the cooperating agent. A good part of the theoretical work of the economist refers to a few types of social institutions. For the paradigm of theoretical economics, the optimizing agent is of interest.

I question, firstly, whether there is coherence in trying to specify an exact field within a human science as the authors did. I do not mean by this that there are no laws in the realm of humanities—at least in the horizon of human experience in which we find ourselves—they not only exist but are studied by philosophers and represented in art¹ throughout the ages. Of the greatest of them, Seneca already treated in his letters to his father-in-law Paulinus:

Most mortals, Paulinus, complain about the malice of Nature, because they are born with the prospect of a short existence and because the years given to them pass quickly and swiftly. So that, with the exception of a few, for the others, it is precisely in the splendor of life that it abandon them. (Sêneca 2008)

The individual’s time is restricted to their always insufficient years; their place in the universe, confined to a single rock amidst the infinitude of the cosmos; bound to the ground by gravity; captive to the surface by the air itself; hostage to the need to consume. However, this consumption I mention does not refer to the idea of an exchange economy that some theorists posit as a natural and spontaneous order independent of social space, historicity, and the specificities of each society. I refer solely to the food that every animal needs to survive. This, indeed, is the consumption that is natural law. The adoption of maximizing behavior as a natural law represents one of several generalizations and extrapolations of this nature that are at the center of the critique of the paradigm, not mathematization itself.

When the authors state that the object of exact methods in economics is “all types of regular and standardized behavior, equal for all individuals within a group,” they fail to see that there are very few of these regular and standardized behaviors. The regular behavior is: man seeks food to survive. However, man knows the equilibrium price of food and will demand more of it if it is below this price is not a natural law. If the mechanism of convergence to macroeconomic equilibrium is entirely anchored in the microfoundation of standardized rational behavior, the macroeconomics of the paradigm already becomes fragile, and any development and conclusion obtained from it, no matter how complex and mathematically advanced, will be biased.

It must be emphasized that the paradigm is not alone in these extrapolations. For example, Keynes standardizes human behavior with the animal spirit; Hayek assumes the invariability of the human spirit and the universality, in space and time, of the laws governing the economy in any society. Barbieri and Feijó (2013) follow the same path by defending the existence of an objective reality that is separate from the subject investigating it:

There is something beyond a purely mental exercise involved in the work of the social scientist, for an objective reality thrives that must be rationally studied. Just as in the natural sciences, in the field of social investigation, one must also separate the knowledge formulated about facts, on the one hand, from external and objective reality itself, on other. With this, the traditional separation between the ideas elaborated about reality and this reality itself also extends to the domain of social science. The separation, usual in the physical sciences, between subject and object, partially violated only in quantum mechanics, also applies to the field of social studies.

The existence of a single objective reality is a presupposition of the positivist approach and an instrument of the method of exact sciences. In them, the researcher adopts a position of neutrality—they are a mere passive observer. Such reality does not exist in social science. The latter admits that reality can be seen from various perspectives, and the social researcher knows that their observation and conclusions are value-laden. For example, to assume the neutrality of Hayek’s thought, who wrote The Road to Serfdom in 1944 under the influence of Nazi-fascist totalitarianism on one side, and the Soviets on the other; or that of Marx, who wrote Das Kapital in the era of panoptic factory-prisons, is concerning.

The guise of exact science undoubtedly provides a creative and elegant disguise for economic science. However, to the extent that this view requires the adoption of unsustainable hypotheses, it is not surprising that, after centuries of development of economic theory, crises still take economists by surprise.

It was so in 2008. The year after the crash, Paul Krugman wrote in his New York Times article:

As I see it, the economics profession went astray because economists, as a group, mistook beauty, clad in impressive-looking mathematics, for truth. […] the central cause of the profession’s failure was the desire for an all-encompassing, intellectually elegant approach that also gave economists a chance to show off their mathematical prowess. Unfortunately, this romanticized and sanitized vision of the economy led most economists to ignore all the things that can go wrong. They turned a blind eye to the limitations of human rationality that often lead to bubbles and busts; to the problems of institutions that run amok; to the imperfections of markets — especially financial markets — that can cause the economy’s operating system to undergo sudden, unpredictable crashes; and to the dangers created when regulators don’t believe in regulation. (Krugman 2009)

This illusion is so ingrained in the paradigm that economists, both in academia and in the market, thought they had all the solutions². Obviously, they did not. Now, suppose my argument up to this point is completely wrong and there is, in fact, an exact core in economic science. Then we have that the paradigm’s theory does not represent it, since, as Krugman points out, it has not been useful for making predictions. Naturally, I do not limit my critique to an instrumentalist position: the paradigm’s theory also fails to explain social phenomena. Take growth theory, for example. For decades, models were developed—we have Solow’s seminal formulation in 1956, still univariate; dynamic optimization with Ramsey-Cass-Koopmans in the 1960s; the attempt to explain it through human capital with Lucas in the 1980s; externalities and then technological innovation with Romer. Despite all the elegance and notation worthy of exact science academic theses, none of these works were able to explain the most basic questions about economic growth, notably the problems known as catching up, falling behind, and forging ahead. Therefore, the paradigm fails both from an instrumentalist and a conventionalist perspective.

I believe Krugman’s critique is very pertinent and answers the initial question well: if economics is a social science, it cannot be investigated from a supposed objective reality, nor deduced from initial conditions and universal laws. In this sense, Herscovici (2002) highlights the limits of Popperian epistemology in economic analysis:

In most cases, to make a law universal, it is necessary to empty it of its historical content. The formalism that results from this operation is characterized by the fact that, if a scientific law wants to explain all possible systems, it is unable to explain any real system, in its historical and social specificities.

These criticisms are not foreign to the authors. Still in the introduction of the work, they try to shield themselves:

It seems naive to imagine that the economic agent is always a subject endowed with full rationality, perfect information, who acts in complete markets (with contingent commodities offered for all possible states of nature), who has clarity of their objectives and who always seeks the best result for themselves in the short term, without worrying about the remote consequences of their choices and without caring about others. Indeed, the economic paradigm has been criticized for the naivety of its behavioral assumptions in its analysis. But, in many cases, the label of naivety better characterizes the situation of those who make this type of criticism than the models of theoretical economists themselves, because the hypotheses listed above, in truth, are just simplifications from introductory textbooks. The frontier of theoretical knowledge in economics has long learned to deal with much bolder and more realistic behavioral hypotheses than these. Thus, current theoretical economists know perfectly well how to model agents with limited rationality, imperfect information, incomplete markets, undecided agents with multiple objectives, and who take into account the well-being of other people.

Well, one moment they say that a few institutions are enough for economists, among them the optimizing agent; another moment they say that such an agent has already been overcome. Worse, stating that “current theoretical economists know how to ” indecision seems doubly incoherent to me. First, because, by definition, there is no such thing as a “perfect model.” Second, indecision, uncertainty, is stochastic by hypothesis. If one assumes uncertainty, one does not perfectly model it. But they go further:

One can also criticize the assumptions of the economic paradigm for considering only the model of perfect competition. Another irrelevant criticism, because, varied (sic) market structures, from perfect competition to monopoly, began to be studied as early as the 19th century, and from the late 1920s theoretical knowledge deepened with the consideration of intermediate structures such as monopolistic competition and oligopoly. The assumptions of perfect competition were questioned and rethought by the influence of the criticism of economists of the so-called Austrian school of economics, but not only by it, and replaced by the view of the economic agent with active behavior in market exploration and discovery of opportunities. The basic model of monopoly was questioned and re-examined, in a dynamic context of the economy, by the renowned Joseph Schumpeter.

Ignoring that Schumpeter was a critic of the paradigm’s methodological individualism, I draw attention to the issue of falsifiability in the field of exact sciences. The authors, while proposing a positivist view of economic science, invalidate it scientifically within Popperian philosophy itself by shielding the paradigm from the principle of falsifiability. According to the authors, one cannot question the adoption of the competitive hypothesis, despite it still being found in manuals and academic production; The problematic known as the Cambridge Controversy which, among other things, criticizes the use of production functions is also unfounded because economics today works with other mathematics — but all microeconomics manuals, from undergraduate to postgraduate, are built on Lagrangian optimization in production functions. How can the law be timeless and change so much? How can it be universal and be only at the frontier of knowledge? Was perfect competition a universal law before and now it is not? Will today’s frontier of knowledge be a universal law or will it have the same fate as the previous ones? By changing the demarcation of hypotheses to defend the theory, the paradigm becomes non-falsifiable and, therefore, ceases to be a scientific theory and becomes dogma.

Very clearly and somewhat contradictorily, the authors place the core of the maximizing equilibrium paradigm, the target of criticism from other schools of thought, as an incontestable truth:

Traditional methodologists are greatly concerned with the empirical confrontation of theory and that the real world can effectively function as the ultimate judge in the validation and eventual discrediting of economic theories. The economic paradigm adheres to a method that indeed gives great importance to empirical evaluation. However, in this regard, two important observations are due: (1) not everything produced by the paradigm should be empirically tested; (2) not all theories are created to be tested. The economic paradigm can, therefore, be conceived as formed by a core containing only contributions of pure mathematics and a peripheral belt of theories that produce empirically testable conclusions.

Not being a subtle contradiction, they try to evade them shortly after, confusing, once again, economics—as a phenomenon that cannot be dissociated from its social specificities—with pure mathematical abstraction:

Radical Popperian methodologists would revolt at the idea of economic theories that never confront empirical evidence. But, in fact, in the mathematical core of the economic paradigm, mathematical economists (or economic mathematicians) work as if they were in the environment of the pure mathematician. No methodologist, however extreme, would consider mathematics innocuous for not confronting empirical facts. Pure mathematical economics should also not be required to build some bridge connecting abstract models with some procedure of verification in observed facts.

The authors frequently compare the economic paradigm with the quantum-relativistic paradigm of modern physics. However, a superficial examination is enough to note the differences: has what is given as theory been abandoned? On the contrary, the body of knowledge is widely known and studied. It is not hidden only “at the frontier of knowledge,” as the authors claim the elements that absolve economic theory from criticism are. The results found by Einstein at the beginning of the 20th century are still used to find bodies in the cosmos, from planets to black holes, and with each experiment they are put to the test and remain capable of explanation and prediction. Falsifiability does not work with absolute truths. Scientific knowledge is momentary and not permanent. A theory can only be qualified as scientific and significant as long as it can be tested and disproven.

But the authors do not allow its contestation, and the entrenchment continues with ad hominem arguments: if an economist is critical of economics as an exact science, then they do not understand mathematics:

Let’s look at the problem of misunderstanding mathematics. Critics of the use of mathematics in economics, in general, are not mathematical economists. Many of them did not have professional training in mathematics. They ventured into other areas. This does not mean they were not good students of mathematics. Marshall, for example, a critic of mathematics in economics, whose use, for him, should not be abusive, proved to be, before concentrating on economics, an excellent mathematician. Even Carl Menger, the father of the Austrian school, totally critical of the use of mathematics in economics, reportedly, was a good student of mathematics in high school. Keynes was a critic of mathematics, yet, a good mathematician, even without having the mind and experience of a professional mathematician. In fact, he graduated, with distinction, in this discipline.

Which is obviously an absurd argument. Imagine criticizing Foucault by saying: “critics of the police state, in general, are not police officers. Many of them did not have professional training in militarism.” And they continue invalidating criticisms of the paradigm:

The contrary view to mathematics of these classic exponents of scientific economics is perfectly understandable. Especially because they were bound by the mathematical development of their time. The problem is when, nowadays, the members of the schools bequeathed by them cling to the same criticisms of the use of mathematics, sustaining them in a crude and outdated view of mathematics. They criticize, for example, that economic variables do not obey functional relationships. But who said that current mathematical economics uses functions? Economists of the paradigm often work with correspondences, maps, and other mathematical techniques much more powerful than functions. They criticize the use of mathematics in economics based on the argument that economic variables do not describe a well-behaved trajectory; so that they cannot be accompanied by continuous and smooth curves, i.e., differentiable at all points. Who said that the current paradigm has to launch the hypothesis of differentiability and that it only uses differential and integral calculus? In fact, current economists of the paradigm work with convex analysis, which dispenses with the differentiability of functions, taking only the much weaker hypothesis of convexity. The existence of continuous curves is no longer imposed. They work with much more general and flexible descriptions, such as upper semicontinuity, etc. The use of calculus in mathematical economics has long been replaced by convex analysis, topology, and differential topology.

Well then, let’s take the preface of an advanced mathematical economics work:

In recent years, the usual optimisation techniques, which have proved so useful in microeconomic theory, have been extended to incorporate more powerful topological and differential methods, and these methods have led to new insights into the qualitative behaviour of general economic systems. These developments have necessarily resulted in an increase in the degree of formalism in the publications in the academic economic theory journals; a formalism which can often deter graduate students. My hope is that the progression of ideas presented here will familiarise the student with the geometric concepts underlying these topological methods, and, as a result, make modern mathematical economics and general equilibrium theory more accessible. (Schofield 2018)

It is not a question of mathematical limitation or an absurd hierarchy of mathematics—as if topology were better than calculus, or something of the sort—but a question of propositions. It does not matter whether the theory studies consumer and firm optimization or Walrasian general equilibrium through topology or calculus, but it is the equilibrium itself, the maximizing behavior itself that is under criticism, and not the mathematics used to demonstrate it.

2 THE PROGRAMMED ECONOMIST

So, why do theorists of the economic science paradigm continue to cling to such problematic concepts? In Sistemas de Ensino e Sistemas de Pensamento, Pierre Bourdieu posits the educational system as one of the most effective instruments for the moral and logical integration of society, which produces the “programmed” individual—homogeneous in perception, thought, and action:

If one admits that culture and, in this particular case, erudite culture in its quality as a common code is what allows all holders of this code to associate the same meaning with the same works and, reciprocally, to express the same significant intention through the same words, the same behaviors, and the same works, one can understand why in terms of participation in a common sense understood as a condition for communication. (Bourdieu 2015)

Perhaps, a similar logic has taken over scientific production. Apparently oblivious to the problem, the authors seem to celebrate homogeneous thought in economic science:

Without explaining it for now, it is sufficient to say that the Kuhnian paradigm of economics is what undergraduate students are obliged to learn in theoretical disciplines. Every macroeconomics student studies textbooks written by authors such as Blanchard, Boyes, Dornbusch, […]. Then it is observed, in Brazil and in the “free world,” that students are subjected to the same training, because, despite differences in approach, style, and depth, all these didactic manuals of economic theory resemble each other in many aspects. In all of them, the institutions of private property, contract, maximizing agent, etc., thrive as presuppositions of theoretical models. All address economic problems with algebra and graphs, and develop equilibrium models. This homogeneous training of students is characteristic of the scientific practice that Kuhn calls normal science. Typical in sciences with consolidated paradigms. The economic paradigm subjects students to the same basic theoretical training. It is clear that similar training leads professionals trained in this environment to act in a standardized way in their work with their theories and in their empirical testing. (Barbieri and Feijó 2013)

If we take poetic license to extend Bourdieu’s theory of fields to the scientific community, we can imagine academic production located in the subfield of erudite production and the editors of journals in the field of power. In order to obtain greater symbolic capital, producers must publish in the most prestigious periodicals. Consequently, production submits to the standardized thought desired by the field of power. Now, the data that allow us to reason in this way are provided by the authors themselves:

[…] in one of the most prestigious economics journals (if not the most prestigious in the world), more than half (54%) of the articles were about mathematical models without any data, almost a quarter of them (24.6%) were about empirical analysis, using statistical inference on published data or based on simulation and artificial experiments. Only 21.4% of the articles in the American Economic Review were dedicated to other topics, of which only 11.6% concerned models without mathematics and without data. Consider the 31 most prestigious international economics journals, which received an A1 classification from CAPES, among the international tops. Twenty of them focus heavily on articles with mathematical theories and eventual econometric tests; five are dedicated to publishing essays in applied economics and two are focused on econometrics (Journal of Applied Econometrics, Journal of Econometrics). Three others are distributed among history of economic thought (History of Political Economy), studies in methodology of economics (Journal of Economic Methodology), and reviews (Journal of Economic Literature). Only a single A1 journal publishes topics from economic schools not inserted in the paradigm (Journal of Post Keynesian Economics). So, of the 31 economics journals classified as A1, there is only one dedicated to a school of economic thought outside the paradigm of economic science, which prioritizes works in the line of so-called post-Keynesianism. It is also considered that studies outside this paradigm, including also Marxian, Austrian, institutional economics, etc., can be accepted for publication in at least three more A1 journals: Cambridge Journal of Economics, Journal of Economic Methodology, and History of Political Economy.

All this points to Bourdieu’s thesis that “molded minds are predisposed to maintain with their peers a relationship of complicity and immediate communication.” It is the common way of approaching common problems. They close themselves off in their common symbols, signs, language, and thought; they close themselves off to approaches and criticisms from those who do not use their common repertoire. And it is through this mechanism that economists continue to be caught off guard by crises that insist on finding them unprepared.

1 Easy installation for RStudio, Quarto and Positron

Right now, there are no official repositories for Rstudio, Quarto or Positron for Linux. This means that users have to download the binaries from their respective websites and install them manually. But I wanted the same experience of macOS users, who can count on Positron software distribution via Homebrew. There are some unofficial PPAs available for Rstudio, but I couldn’t find any for Quarto or Positron, So I’ve decided to make my own and share with you!

What this project do is scraping the urls for the latest versions of Rstudio, Quarto and Positron from their respective websites or Github releases, download the files, upload them to Sourceforge and finally create a PPA that can be used to install these tools on Ubuntu, Fedora and other Linux distributions.

This two scripts holds the logic to create the PPA:

1. Define the latest versions of RStudio, Quarto and Positron.
2. Download the binaries from their respective websites or Github releases.
3. Upload the binaries to Sourceforge.
4. Generate the metadata files required for the PPA.
5. Create the release files for the PPA.
6. Generate checksums for the files.s

2 Support

• .deb packages for Ubuntu and Debian-based distributions.
  • Rstudio (amd64)
  • Quarto (amd64 and arm64)
  • Positron (amd64 and arm64)
• .rpm packages for Fedora and Red Hat-based distributions.
  • Rstudio (x86_64)
  • Positron (x86_64 and aarch64)
  • Quarto (x86_64)

3 Debian/Ubuntu Installation

To enable this repository and install the latest RStudio, Quarto, or Positron:

1. Add the repository (amd64):

   echo "deb [trusted=yes] https://downloads.sourceforge.net/project/r-tools-ppa/deb_amd64 stable main" | sudo tee /etc/apt/sources.list.d/r_tools_ppa.list
   sudo apt update

   Or for arm64:

   echo "deb [trusted=yes] https://downloads.sourceforge.net/project/r-tools-ppa/deb_arm64 stable main" | sudo tee /etc/apt/sources.list.d/r_tools_ppa.list
   sudo apt update

2. Install a package (e.g., RStudio):

   sudo apt install rstudio

4 Fedora/Red Hat Installation

To enable this repository and install the latest RStudio, Quarto, or Positron:

1. Download the repository file:

   sudo curl -L -o /etc/yum.repos.d/r_tools_ppa.repo \
     https://sourceforge.net/projects/r-tools-ppa/files/rpm_x86_64/repo.txt/download

   Or for aarch64 architecture:

   sudo curl -o /etc/yum.repos.d/r_tools_ppa.repo \
     https://sourceforge.net/projects/r-tools-ppa/files/rpm_aarch64/repo.txt/download

2. Update & install (e.g., RStudio):

   sudo dnf update
   sudo dnf install rstudio

1 LET’S GET SOME DATA

Have you ever heard about presidents dataset? From the built-in {datasets}, it stores US presidents quarterly approval ratings from 1945 to 1974. Does it hold any relationship with consumer price index (CPI), real GDP, unemployment rate or interest rates? Let’s find out!

# convert to dataframe
presi <- data.frame(
  date = zoo::as.Date(zoo::as.yearqtr(time(presidents))),
  ratings = as.vector(presidents)
) |>
  na.omit()

# get CPI and funds rates from FRED
quantmod::getSymbols(
  c("CPIAUCSL", "FEDFUNDS", "GDPC1", "UNRATE"),
  src = "FRED"
)

[1] "CPIAUCSL" "FEDFUNDS" "GDPC1"    "UNRATE"  

# convert to time series
series <- lapply(list(CPIAUCSL, FEDFUNDS, GDPC1, UNRATE), function(serie) {

  # serie name
  serie_name <- names(serie)
  # convert to dataframe
  data.frame(
    date = zoo::as.Date(zoo::as.yearmon(time(serie))),
    value = as.vector(serie)
  ) |>
    setNames(c("date", serie_name))
})

# merge datasets
presi_merged <- Reduce(
  function(x, y) merge(x, y, by = "date"),
  c(list(presi), series)
)

# visualize
kbl(head(presi_merged), booktabs = TRUE, digits = 2) |>
  kable_styling(latex_options = c("striped", "scale_down"))

# plot
presi_merged |>
  tidyr::pivot_longer(-date) |>
  ggplot(aes(x = date, y = value)) +
  geom_line() +
  scale_x_date(date_labels = "%Y") +
  facet_wrap(~name, nrow = 5, scales = "free_y")

2 ERROR #1: NOT TESTING SIGNIFICANCE

The first, and often only, resource for variable selection and model specification among those not initiated in time series analysis is the Pearson correlation coefficient . If we try to use it, we’ll’ have the following results (Table 2):

# Pearson correlation
cor(subset(presi_merged, select = -date)) |>
  kbl(booktabs = TRUE, digits = 2) |>
  kable_styling(latex_options = c("striped", "scale_down"))

According to the rule of thumb for interpreting the effect size of the Pearson correlation coefficient¹, CPI, interest rates and GDP have all large correlation with presidential approval ratings, while unemployment rate has small correlation.

Even if it were appropriate to use the Pearson correlation coefficient to select variables, a test must be performed to verify if the found coefficient is statistically significant. In this case, we can verify that the 10% correlation between presidential approval and unemployment rate is not significant. Ignoring significance tests results in erroneous interpretation, leading the analyst to find relationships where, in fact, none exist.

# p-values for t-test
sapply(subset(presi_merged, select = -c(date, ratings)), function(serie) {
  cor.test(presi_merged$ratings, serie)$p.value
}) |>
  kbl(booktabs = TRUE, digits = 2) |>
  kable_styling(latex_options = c("striped", "scale_down"))

When it comes to inferential statistics, making any statement requires performing hypothesis tests. Except in extreme cases that prevent the estimation of some coefficient, in most cases, it is possible to obtain some statistical measure, such as in regressions, analysis of variance, correlations, etc. However, obtaining a number does not mean it is valid or can be used.

The correlation coefficient calculated from a sample () is a point estimate of a population parameter (). Like any sample-based estimate, it has sampling variability and, therefore, must be accompanied by a confidence interval (CI) that reflects the uncertainty of this estimate and provides a range of plausible values for the true population parameter. This value is a central point (mean) around which the confidence interval for the true value is constructed.

If, due to sampling variability, (e.g., with a confidence interval of ), then there is no statistical evidence to claim that the mean value of 0.05 is different from zero.

3 ERROR #2: USING THE PEARSON CORRELATION COEFFICIENT

In section Section 2, I said “even if it were appropriate,” because it is not. The Pearson correlation is inadequate for assessing the correlation between two time series because it violates fundamental assumptions of the Pearson coefficient.

First, it violates the independence of observations. One of the assumptions of is the independence of observations within each variable and between variables. Time series, by nature, usually exhibit autocorrelation, that is, an observation is correlated with .

Second, it ignores the relationship between lags. The correlation between two time series and may not be contemporaneous. One series may influence the other with some delay (e.g., correlated with but not with ). The Pearson correlation only measures the linear relationship at the same point in time, ignoring these lagged relationships.

There are some valid ways to identify the relationship between two time series. One of the simplest is the cross-correlogram analysis. It allows you to identify not only the contemporaneous relationship but also the relationship at each lag of the time series.

Now let’s look at the cross-correlogram between the time series of unemployment rate and presidential approval ratings:

ccf(
  presi_merged[["UNRATE"]],
  presi_merged[["ratings"]],
  main = ""
)

If it were appropriate to analyze the two time series directly, we would see that, although there is no significant contemporaneous correlation — as we see by the correlation within the confidence interval at lag 0 —, they appear from , which would mean that unemployment rate from 2 months ago impacts the current presidential approval ratings.

1 Hierarchical and grouped time series

Hierarchical time series are those that can be naturally aggregated or disaggregated in a nested structure (R. J. Hyndman and Athanasopoulos 2021). To illustrate, take the Brazilian GDP series. It can be disaggregated by state, which in turn can be disaggregated by municipality.

This structure can be represented by equations for any level of aggregation.

Thus, the national aggregate can be represented only by the state aggregates, as in (1), or as the aggregate of municipalities (2). The aggregate for the state of Espírito Santo is represented by (3).

Alternatively, we can describe the complete structure in matrix form:

On the other hand, GDP can also be disaggregated in a cross-sectional way according to economic activity—crops, livestock, manufacturing, extractive industry, capital goods, intermediate goods, clothing trade, automotive, services, etc. This structure cannot be naturally disaggregated in a single way, as in the hierarchy of states and municipalities. It cannot be nested by a single attribute like geography. This type of structure is called grouped series.

Combining both, we have the structure of hierarchical grouped series. Unlike the hierarchical structure, which can only be aggregated in one way—as with municipalities under states—the addition of the grouped structure can occur either above (Figure 3) or below (Figure 4) the hierarchy.

In matrix notation, the structure of Figure 4 is represented as below. Formally, the left-hand side of the equation is composed of the -dimensional vector with all observations at time for all hierarchy levels. The right-hand side is composed of the summing matrix of dimension that defines the equations for every aggregation level, and the matrix composed of the series at the most disaggregated level.

2 Top-down and bottom-up approaches

Perhaps the most intuitive ways to think about forecasts for these types of structures are the top-down and bottom-up approaches. Take the structure described in Figure 1, for example. We can forecast for the horizon of the aggregate Brazilian GDP, represented at the top of the hierarchy by Total (6), and then distribute the forecasted values proportionally among states and municipalities.

This is the top-down approach. In it, the forecasts for the most disaggregated levels of the hierarchy are determined by a proportion of the aggregate level. For example, the forecasts for Vitória are given by equation (7).

For this, we have to define a matrix with all these weights, which, following the formulation of R. J. Hyndman and Athanasopoulos (2021), we will call :

is an matrix that multiplies the matrix , which in turn is composed of the base forecasts—the individual forecasts for all aggregation levels. The equation for the top-down approach will then be:

In matrix notation for the structure of Figure 1, we have:

Which gives us a proportion of the total for each element at the most disaggregated level.

Replacing the matrix , we have the equations that define each forecast in the structure as a function of proportions of the aggregate forecast.

The bottom-up approach, on the other hand, starts from the opposite reasoning and defines the forecasts for each element of the structure from the forecasts of the most disaggregated elements. For this, just modify the matrix .

Which results in the desired equations. Therefore, defines the approach—whether top-down or bottom-up—and defines how the forecasts are summed to form the forecast equations for each element of the structure.

3 Coherence and reconciliation

Whether by summing the forecasts from the most disaggregated level to form the upper levels of the hierarchy (bottom-up) or proportionally distributing the forecasts from the most aggregated level (top-down), the vector represents the coherent forecasts. This means that the forecasts “match”, i.e., they are correctly totaled—the forecasts for each aggregate element correspond to the sum of the forecasts for the lower levels of the hierarchy. This is ensured by the multiplication of the matrices .

Without this pre-multiplication, nothing would guarantee the coherence of the forecasts. Taking the structure of Figure 1 as an example, it would be an unlikely coincidence for the forecasts of the aggregate for the state of Espírito Santo to be exactly the sum of the individual forecasts of its municipalities. This is because each series may follow a different process (e.g., ARIMA) with different errors and variances.

The methods for generating coherent forecasts from base forecasts are called reconciliation methods. The traditional reconciliation methods presented, top-down and bottom-up, use limited information. In the top-down method, only information from the most aggregated level is used—which is why only the first column in (8) is nonzero. In the bottom-up approach, only information from the most disaggregated levels is used, resulting in the identity submatrix in (13), while the columns representing the most aggregated levels are zero.

Alternatively, we can think of any matrix that uses all available information and has some properties that ensure that the coherent forecasts have the smallest possible error. This is the research problem addressed in optimal reconciliation, and it is a relatively new topic. That is, point forecasts for hierarchical time series are an old thing. At least since the 1970s, research has been published on bottom-up and top-down approaches, their advantages and disadvantages, and attempts to define the best method¹. However, it is only in R. J. Hyndman et al. (2011) that a practical approach is formalized that uses all available information (i.e., the forecasts of all elements at all levels of the hierarchy) by estimating the matrix via generalized least squares (GLS) regression.

However, to be able to estimate the model by GLS, the variance-covariance matrix of the errors is needed. R. J. Hyndman et al. (2011) use the coherence error matrix, i.e., the difference between the reconciled forecasts and the base forecasts, which is rank-deficient and unidentified and therefore cannot be estimated. The authors circumvent this problem by adopting, instead of the variance-covariance matrix of the errors, a constant diagonal matrix, i.e., they assume constant variance of the reconciliation errors, which ends up falling into the estimation of by ordinary least squares (OLS).

Estimation by this method results in an optimal reconciliation that depends only on the matrix , i.e., the hierarchical structure, and is independent of the variance and covariance of the base forecasts —which is not a satisfactory conclusion.

Rob J. Hyndman, Lee, and Wang (2016) attempt to improve the method by using the estimated variances of the base forecasts (within-sample) as an estimate for the variance-covariance matrix of the reconciliation errors, so as to use them as weights and perform optimal reconciliation by weighted least squares (WLS). Thus, more accurate base forecasts have greater weight than noisier ones. However, they do not provide a theoretical justification for using the diagonal of the variance-covariance matrix of .

Wickramasuriya, Athanasopoulos, and Hyndman (2019) argue that what really matters is that the reconciled forecasts have the smallest error—it does not matter that the reconciled forecast is closer to the base forecast, but that the reconciled forecasts are as accurate as possible, since that is what will be used! So, they correct the optimal reconciliation approach to the objective of minimizing the errors of the reconciled forecasts , instead of the reconciliation errors of the base forecasts . Since this implies minimizing the variance of , i.e., minimizing the sum of the diagonal, the trace, of the variance-covariance matrix of , they called this method Minimum Trace (MinT). In parallel, they use the triangle inequality to show that the reconciled forecasts obtained by this method are at least as good as the base forecasts.

And, from 2021 onwards, there have been interesting developments in the probabilistic extension of this theoretical body (and this is where I am conducting my master’s research). Panagiotelis et al. (2021) reinterpret the literature on coherence and reconciliation of point forecasts from a geometric approach, providing alternative proofs for previous conclusions while also providing new theorems. In addition, Panagiotelis et al. (2021) extend this geometric interpretation to the probabilistic context, providing parametric and non-parametric (via bootstrapping) methods for reconciliation of probabilistic forecasts, i.e., to reconcile forecasts obtained from the entire distribution, not just the mean.

4 Anyway…

If you made it this far, I recommend you find something more interesting to do with your life. In the next post (maybe) I’ll show how to do all this in practice with our beloved #RStats.

1 Motivation

It crossed my mind an analogy about learning a foreign language: it’s impossible to learn a new language by translating word by word. It’s not only a matter of vocabulary. I mean, each language has it’s own grammar, phrasal verbs, diction, expressions, pace etc. That kind of issue also appears when learning a new programming language and I think importing packages is a good, yet very simple, example of that.

2 Calling A Function From A Package

# search for machine learning measures that contais "AUC" in the {mlr3} package
mlr3::mlr_measures$keys("auc")

[1] "classif.auc"       "classif.mauc_au1p" "classif.mauc_au1u"
[4] "classif.mauc_aunp" "classif.mauc_aunu" "classif.mauc_mu"  
[7] "classif.prauc"    

# tired: explicitly calling from {ggplot2}
t1 = mtcars |>
  dplyr::mutate(hp_by_cyl = hp / cyl)

# wired: attaching {ggplot2} namespace
library(dplyr)

Anexando pacote: 'dplyr'

Os seguintes objetos são mascarados por 'package:stats':

    filter, lag

Os seguintes objetos são mascarados por 'package:base':

    intersect, setdiff, setequal, union

t2 = mtcars |>
  mutate(hp_by_cyl = hp / cyl)

# are they equivalent?
all.equal(t1, t2)

[1] TRUE

# detaching dplyr
detach("package:dplyr")

# attaching only mutate():
library(dplyr, include.only = "mutate")

# inspecting modules in {pandas}
import pandas
dir(pandas)

['ArrowDtype', 'BooleanDtype', 'Categorical', 'CategoricalDtype', 'CategoricalIndex', 'DataFrame', 'DateOffset', 'DatetimeIndex', 'DatetimeTZDtype', 'ExcelFile', 'ExcelWriter', 'Flags', 'Float32Dtype', 'Float64Dtype', 'Grouper', 'HDFStore', 'Index', 'IndexSlice', 'Int16Dtype', 'Int32Dtype', 'Int64Dtype', 'Int8Dtype', 'Interval', 'IntervalDtype', 'IntervalIndex', 'MultiIndex', 'NA', 'NaT', 'NamedAgg', 'Period', 'PeriodDtype', 'PeriodIndex', 'RangeIndex', 'Series', 'SparseDtype', 'StringDtype', 'Timedelta', 'TimedeltaIndex', 'Timestamp', 'UInt16Dtype', 'UInt32Dtype', 'UInt64Dtype', 'UInt8Dtype', '__all__', '__builtins__', '__cached__', '__doc__', '__docformat__', '__file__', '__git_version__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', '_built_with_meson', '_config', '_is_numpy_dev', '_libs', '_pandas_datetime_CAPI', '_pandas_parser_CAPI', '_testing', '_typing', '_version_meson', 'annotations', 'api', 'array', 'arrays', 'bdate_range', 'col', 'compat', 'concat', 'core', 'crosstab', 'cut', 'date_range', 'describe_option', 'errors', 'eval', 'factorize', 'from_dummies', 'get_dummies', 'get_option', 'infer_freq', 'interval_range', 'io', 'isna', 'isnull', 'json_normalize', 'lreshape', 'melt', 'merge', 'merge_asof', 'merge_ordered', 'notna', 'notnull', 'offsets', 'option_context', 'options', 'pandas', 'period_range', 'pivot', 'pivot_table', 'plotting', 'qcut', 'read_clipboard', 'read_csv', 'read_excel', 'read_feather', 'read_fwf', 'read_hdf', 'read_html', 'read_iceberg', 'read_json', 'read_orc', 'read_parquet', 'read_pickle', 'read_sas', 'read_spss', 'read_sql', 'read_sql_query', 'read_sql_table', 'read_stata', 'read_table', 'read_xml', 'reset_option', 'set_eng_float_format', 'set_option', 'show_versions', 'test', 'testing', 'timedelta_range', 'to_datetime', 'to_numeric', 'to_pickle', 'to_timedelta', 'tseries', 'unique', 'util', 'wide_to_long']

# what is attached into the symbol table?
print(*globals(), sep = "\n")

__name__
__doc__
__package__
__loader__
__spec__
__builtins__
r
pandas

# import pandas
import pandas

# what is attached into the symbol table?
print(*globals(), sep = "\n")

__name__
__doc__
__package__
__loader__
__spec__
__builtins__
r
pandas

# this will result in a NameError: name 'DataFrame' is not defined
DataFrame(
  {
    "capital": ["Vitoria", "São Paulo", "Rio de Janeiro"],
    "state": ["Espírito Santo", "São Paulo", "Rio de Janeiro"]
  }
)

NameError: name 'DataFrame' is not defined

# this will work
pandas.DataFrame(
  {
    "capital": ["Vitoria", "São Paulo", "Rio de Janeiro"],
    "state": ["Espírito Santo", "São Paulo", "Rio de Janeiro"]
  }
)

          capital           state
0         Vitoria  Espírito Santo
1       São Paulo       São Paulo
2  Rio de Janeiro  Rio de Janeiro

# importing entire {pandas} into symbol table
from pandas import *

# the updated symbol table
print(*globals(), sep = "\n")

__name__
__doc__
__package__
__loader__
__spec__
__builtins__
r
pandas
NA
ArrowDtype
BooleanDtype
Categorical
CategoricalDtype
CategoricalIndex
DataFrame
DateOffset
DatetimeIndex
DatetimeTZDtype
ExcelFile
ExcelWriter
Flags
Float32Dtype
Float64Dtype
Grouper
HDFStore
Index
IndexSlice
Int8Dtype
Int16Dtype
Int32Dtype
Int64Dtype
Interval
IntervalDtype
IntervalIndex
MultiIndex
NaT
NamedAgg
Period
PeriodDtype
PeriodIndex
RangeIndex
Series
SparseDtype
StringDtype
Timedelta
TimedeltaIndex
Timestamp
UInt8Dtype
UInt16Dtype
UInt32Dtype
UInt64Dtype
api
array
arrays
bdate_range
col
concat
crosstab
cut
date_range
describe_option
errors
eval
factorize
from_dummies
get_dummies
get_option
infer_freq
interval_range
io
isna
isnull
json_normalize
lreshape
melt
merge
merge_asof
merge_ordered
notna
notnull
offsets
option_context
options
period_range
pivot
pivot_table
plotting
qcut
read_clipboard
read_csv
read_excel
read_feather
read_fwf
read_hdf
read_html
read_iceberg
read_json
read_orc
read_parquet
read_pickle
read_sas
read_spss
read_sql
read_sql_query
read_sql_table
read_stata
read_table
read_xml
reset_option
set_eng_float_format
set_option
show_versions
test
testing
timedelta_range
to_datetime
to_numeric
to_pickle
to_timedelta
tseries
unique
wide_to_long

# and now this works
DataFrame(
  {
    "capital": ["Vitoria", "São Paulo", "Rio de Janeiro"],
    "state": ["Espírito Santo", "São Paulo", "Rio de Janeiro"]
  }
)

          capital           state
0         Vitoria  Espírito Santo
1       São Paulo       São Paulo
2  Rio de Janeiro  Rio de Janeiro

# detaching {pandas}
for name in vars(pandas):
    if not name.startswith('_'):
        del globals()[name]

KeyError: 'annotations'

# attaching only DataFrame()
from pandas import DataFrame

# the updated symbol table
print(*globals(), sep = "\n")

__name__
__doc__
__package__
__loader__
__spec__
__builtins__
r
pandas
NA
ArrowDtype
BooleanDtype
Categorical
CategoricalDtype
CategoricalIndex
DataFrame
DateOffset
DatetimeIndex
DatetimeTZDtype
ExcelFile
ExcelWriter
Flags
Float32Dtype
Float64Dtype
Grouper
HDFStore
Index
IndexSlice
Int8Dtype
Int16Dtype
Int32Dtype
Int64Dtype
Interval
IntervalDtype
IntervalIndex
MultiIndex
NaT
NamedAgg
Period
PeriodDtype
PeriodIndex
RangeIndex
Series
SparseDtype
StringDtype
Timedelta
TimedeltaIndex
Timestamp
UInt8Dtype
UInt16Dtype
UInt32Dtype
UInt64Dtype
api
array
arrays
bdate_range
col
concat
crosstab
cut
date_range
describe_option
errors
eval
factorize
from_dummies
get_dummies
get_option
infer_freq
interval_range
io
isna
isnull
json_normalize
lreshape
melt
merge
merge_asof
merge_ordered
notna
notnull
offsets
option_context
options
period_range
pivot
pivot_table
plotting
qcut
read_clipboard
read_csv
read_excel
read_feather
read_fwf
read_hdf
read_html
read_iceberg
read_json
read_orc
read_parquet
read_pickle
read_sas
read_spss
read_sql
read_sql_query
read_sql_table
read_stata
read_table
read_xml
reset_option
set_eng_float_format
set_option
show_versions
test
testing
timedelta_range
to_datetime
to_numeric
to_pickle
to_timedelta
tseries
unique
wide_to_long
name

1 WHAT’S SO GOOD ABOUT IT THO 🤔?

Most features implemented by R and Language Server extensions are detailed in Kun Ren’s post, so, if you didn’t read that yet, just GO! Here’s some stuff I like about it.

2 SETTINGS

To get started you only need VSCode and R and R-LSP extensions. But here I share the setup that I think provides the best experience:

1. Visual Studio Code and extensions:

• VSCode: the IDE
• vscode-R: R language support
• vscode-r-lsp: R Language Server Protocol Client for VSCode ¹

2. Packages:

• languageserver (Lai 2020): R Language Server Protocol implementation
• httpgd (Rupprecht 2021) (recommended): asynchronous http server graphic device for R

3. Python & Radian:

• Python (recommended): requirement for Radian, implemented in Python
• Radian (recommended): alternative R console, with multi line editing and rich syntax highlighting

4. Pandoc: Universal document converter, responsible for converting R Markdown (.Rmd) files to html. You didn’t need to install it before because the RStudio installer comes with a version of Pandoc included.

3 FIRST STEPS

With VSCode installed, you’ll see the welcome screen, which looks like this:

The first thing to note is that the RStudio project concept (.Rproj) does not exist in VSCode. You have a folder and that’s it, all relative paths point to it. Folder-specific settings can be defined in a .json file in the directory.

In the sidebar, the explorer tab will give you the option to open a folder (a project) or clone a repository. Opening my Advent of Code 2020 folder, you will see my directories and files on the left. There is code in Python and in R. I can open it and organize it any way I want. The terminals are below and they can also be moved elsewhere.

Extensions can be installed from the tab on the left. VSCode itself will recommend installing extensions when you open some kind of file that doesn’t have an interpreter or formatter installed, for example.

Important: do not install the RTools extension together with the extensions mentioned above, as they are not complementary and will cause conflicts.

With the extensions installed, let’s go to settings. First, open the command palette with ctrl+shift+p. There you’ll be able to access several commands and shortcuts in VSCode, such as:

• new terminal (R, Python etc)
• launch RStudio addins
• compile RMarkdown files (knit .Rmd)
• access settings and shortcut keys
• install packages
• format files and MANY others

Accessing the VSCode settings, we can configure both the editor and its extensions:

Here are the ones I currently use. Change as needed and according to your preference. Obs.: by the time you’re reading this they may be changed, since R support is on active development.

{
    // VSCode setup
    "terminal.integrated.shell.windows": "C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "editor.formatOnPaste": true, // recommended
    "editor.formatOnType": true, // recommended
    "diffEditor.ignoreTrimWhitespace": false,
    "files.autoSave": "off",

    // Python setup
    "python.pythonPath": "C:\\...\\anaconda3\\python.exe",
    "jupyter.alwaysTrustNotebooks": true,

    // R setup
    "[r]": { // required
        "editor.defaultFormatter": "REditorSupport.r-lsp"
    },
    "files.associations": { // required
        "*.rmd": "markdown",
        "*.Rmd": "rmd"
    },
    "r.sessionWatcher": true, // default
    "r.bracketedPaste": true, // required
    "r.rterm.windows": "C:\\...\\anaconda3\\Scripts\\radian.exe", // for Radian
    "r.rpath.windows": "C:\\...\\R\\R-4.1.0\\bin\\R.exe", // required
    "r.lsp.debug": true, // required
    "r.alwaysUseActiveTerminal": true, // required
    "r.rtermSendDelay": 0,
}

And a few tweaks on your .Rprofile:

# options
options(
  # activate RStudio Addins on command pallet
  vsc.rstudioapi = TRUE,
  # interactive plots with {httpgd}
  vsc.use_httpgd = TRUE,
  # radian highlight scheme (choose what suits you)
  radian.color_scheme = "native",
  # code completion triggers
  languageserver.server_capabilities = list(
    signatureHelpProvider = list(triggerCharacters = list("(", ",", "$")),
    completionProvider = list(
      resolveProvider = TRUE, triggerCharacters = list(".", ":", "$")
    )
  )
)

Important: not installing {httpgd} or not including this setting in .Rprofile will cause your graphics to be plotted in a fixed-size window, without the possibility of manipulating the plot size interactively.

4 AT LAST

And that’s it for the basics! There’s still a lot of things to explore but you can already get an idea of the tool’s potential :p

1 THE FITTING PROCESS AND OVERFITTING

Consider a fitting function , a set of points with , decision variables or parameters and output . Unlike the classical approach, where, in the case of the classical linear regression model, there is a theoretical model with coefficients estimated by ordinary least squares (OLS) that is guaranteed by the Gauss-Markov theorem to be the best linear unbiased estimator (BLUE), in machine learning the goal is to iteratively find a meta-model that best approximates the function using the information contained in , that is, we want to fit a regression function to our data so that has the smallest approximation error .

To check how well the model approximates the real function , we need a loss function which, in the case of regression, will be the quadratic loss or the absolute loss . These values are averaged to form the cost functions mean squared error (MSE) and mean absolute error (MAE).

Given the loss function, we can define the risk associated with the model fitting function as where is the joint probability density function. Since we do not have the real function but seek an estimated function that approximates it, we have which is the generalization error or conditional risk associated with the predictor.

Thus, we can estimate the generalization error of the model. Since we do not know the distribution , we replace it with the test sample and get If we replace the test sample with the training sample used to fit the model, we have the so-called resubstitution error Naturally, in this case we would be using the training data both to train the predictor and to estimate the generalization error, which would lead to a biased estimate of the generalization error. If we used this estimate for model selection, this bias would favor models more adapted to the sample.

The problem is that in these models, given enough iterations, the resubstitution error tends to zero. This happens because as the predictor adapts more and more to the training data, it will memorize the relationship between the set of points and the output , that is, it will fit perfectly to the shape of the function to be modeled. And a perfectly fitted model does not necessarily translate into the ability to predict future (out-of-sample) data.

In general, it is expected that the predictor reduces its bias during training just enough to be able to generalize its prediction to out-of-sample data at an optimal level of accuracy. Beyond this point, reducing bias is penalized by increased variance, i.e., by reducing its ability to predict future data. This process is called overfitting. This means we cannot consider the predictor’s performance on if we want to honestly estimate the real performance of the model.

2 RESAMPLING

One way to correct this problem is to split the sample into a training set and a test set such that and . Thus, we can train the model on to obtain and calculate its generalization error using the data from . This approach is called hold-out and it is simple to implement and use, since the observations in the test set are completely independent from those used to train the model. The estimate of the generalization error then becomes Two problems remain:

1. A large sample is required, since there must be enough data in both the training set to fit an adequate model and in the test set to perform a statistically valid performance evaluation.

2. This method is not sufficient to detect variance and instabilities in the training sample. More complex models, especially nonlinear ones, can produce very different results with small changes in the training data.

It is precisely to deal with these situations that resampling techniques were developed. All these techniques repeatedly generate training subsets and test subsets from the available dataset, fit a model with each training set, and assess its quality on the corresponding test set. The estimate of the generalization error then becomes The generalization error given in equation (1) depends on both the size of the sample used to train and to test the fitted model. Therefore, we must ensure that the sample size used to check the generalization error of a model estimated from data points is close to . If, for example, the training set is much smaller than the total sample, the error will be overestimated, since much less information was used to calculate the estimator.

Similarly, the quality of the generalization error estimator obtained in (2) from a resampling strategy also depends greatly on the size of the sets relative to the original sample, the number of subsets used, and the dependency structure between the subsets —again, more complex models are more sensitive to changes in the dataset and the variance between subsets tends to be higher. The estimator’s error is usually measured by the mean squared error (MSE): This estimator can also be represented as the sum of the squared bias and the variance: Where the bias expresses the average difference between an estimator and the true value, while the variance measures the average dispersion of the estimator. These quantities are defined as follows: and

3 TO CONCLUDE

With this introduction, I hope the concepts have become clearer and that it has helped a bit in understanding meta-modeling. I’m sure I won’t see the results the same way the next time I apply resampling techniques for selection and tuning of my models!

1 INTRODUCTION

The standard workflow of a machine learning project consists of:

1. splitting your sample into training and test sets;
2. choosing the appropriate algorithm¹ for the task type;
3. passing the training sample to the algorithm to create a model of the relationship between the response variable (output features) and the explanatory variables (input features);
4. passing the test data to the trained model to produce predictions;
5. comparing the predictions with the sample data;
6. measuring the model’s performance using established accuracy metrics.

The process of repeating this workflow several times, splitting the training sample into different parts and using others as fake test samples, is called resampling, a vital process for the calibration stage and to avoid overfitting.

Depending on the data, the type of task, and the chosen algorithm, several filters may be necessary, such as normalization, feature selection, and handling outliers or missing data. For these cases, {mlr3} has new solutions that stand out not only compared to its predecessor {mlr} but also to other machine learning frameworks in R, such as {caret} and {tidymodels}.

2 STRAIGHT TO THE POINT

To get to know the package’s basic functionalities, we’ll use one of the datasets included in R, swiss. This dataset consists of standardized measurements of fertility and socioeconomic indicators for 47 Swiss provinces in 1888.

# creating dataframe
data = swiss

# overview
skimr::skim(data)

Among the available variables, we can choose to model infant mortality Infant.Mortality based on the other features, which are:

• Fertility: Fertility measure. Like infant mortality, it is scaled between 0-100.
• Agriculture: Percentage of men involved in agriculture as an occupation.
• Examination: Percentage of conscripts well evaluated in army exams.
• Education: Percentage of conscripts with education above primary.
• Catholic: Percentage of Catholics (as opposed to Protestants).

The workflow starts with the creation of the task, which is an object that contains the data and information about the task to be performed, such as the response variable² and the other features, as well as their types. Since we want to predict a continuous numeric variable, this is a regression task.

# importing package
library(mlr3verse)

# creating task
task_swiss = TaskRegr$new(
  id = "swiss",
  backend = data,
  target = "Infant.Mortality"
)

Notice anything unusual? Since {mlr3} works with the R6 class, its handling is more similar to other object-oriented languages, such as Python. This class has two special properties:

• Methods belong to objects and are called as object$method() and not as generic functions like foo(). This is the OOP (object-oriented programming) paradigm. In the example above, there is no function to create a task like task_regr_new(), but a new() method associated with the TaskRegr object;

• R6 class objects are mutable, i.e., they are modified in place (modify-in-place) and therefore have reference semantics. This means they are not copied with each modification, unlike regular data frames (S3 class), which is a factor in memory allocation and, consequently, speed.

The side effect is that this is not very familiar to people who only know R and at first it may seem unnatural and confusing.

Once the object is created, we can access it to check and visualize the information it contains:

# checking
task_swiss

── <TaskRegr> (47x6) ───────────────────────────────────────────────────────────
• Target: Infant.Mortality
• Properties: -
• Features (5):
  • dbl (3): Agriculture, Catholic, Fertility
  • int (2): Education, Examination

# visualizing
autoplot(task_swiss, type = "pairs")

We can see that only fertility is linearly correlated with infant mortality—the higher the fertility, the higher the mortality—and we can expect it to have more weight in predictions. The other variables do not show significant linear correlation with the response variable. However, they show moderate or strong correlation among themselves, but not enough to present collinearity, which would require treatment.

Now we select the algorithm³ that will be used to train the model. Here I chose XGBoost. The full list can be accessed in this static list, in this dynamic list, or via the function mlr3extralearners::list_mlr3learners(). As mentioned earlier, algorithms are not implemented by the {mlr3} ecosystem, and the lists show the packages where the algorithms were implemented and that must be installed for use.

# defining the learner
l_xgboost = lrn("regr.xgboost")

# checking
l_xgboost

── <LearnerRegrXgboost> (regr.xgboost): Extreme Gradient Boosting ──────────────
• Model: -
• Parameters: nrounds=1000, nthread=1, verbose=0, verbosity=0
• Validate: NULL
• Packages: mlr3, mlr3learners, and xgboost
• Predict Types: [response]
• Feature Types: logical, integer, and numeric
• Encapsulation: none (fallback: -)
• Properties: hotstart_forward, importance, internal_tuning, missings, offset,
validation, and weights
• Other settings: use_weights = 'use'

Let’s understand what the l_xgboost object tells us.

1. Model: Empty, as there is no trained model yet;
2. Parameters: The hyperparameters to be chosen and tuned for model performance;
3. Packages: The package where the algorithm was implemented and from which it will be imported by {mlr3};
4. Predict Type: If response, the prediction is returned as 0 or 1 for classification, or as a value for the response variable for regression—in this case, it will be infant mortality scaled in the [1, 100] range. If “prob”, for classification, the prediction returns the probability between 0 and 1;
5. Feature Type: The types of variables the algorithm can handle. For XGBoost, for example, only numeric variables can be used. This means that factors must be converted into binary values (i.e., 0 or 1), that is, the matrix must be made sparse—for a factor sex, for example, in the preprocessing phase columns sex.male and sex.female would be created, each with values 1 or 0;
6. Properties: Additional properties and capabilities of the algorithm. In this case, XGBoost can compute and return feature importance values for the model; handle missing data; and compute and return feature weights.

As you can see in parameters, no hyperparameters are set. We can access them as follows:

# accessing hyperparameters
head(as.data.table(l_xgboost$param_set))

                  id    class lower upper               levels nlevels
              <char>   <char> <num> <num>               <list>   <num>
1:             alpha ParamDbl     0   Inf               [NULL]     Inf
2:     approxcontrib ParamLgl    NA    NA           TRUE,FALSE       2
3:        base_score ParamDbl  -Inf   Inf               [NULL]     Inf
4:           booster ParamFct    NA    NA gbtree,gblinear,dart       3
5:         callbacks ParamUty    NA    NA               [NULL]     Inf
6: colsample_bylevel ParamDbl     0     1               [NULL]     Inf
   is_bounded special_vals   default storage_type    tags
       <lgcl>       <list>    <list>       <char>  <list>
1:      FALSE    <list[0]>         0      numeric   train
2:       TRUE    <list[0]>     FALSE      logical predict
3:      FALSE    <list[0]>       0.5      numeric   train
4:       TRUE    <list[0]>    gbtree    character   train
5:      FALSE    <list[0]> <list[0]>         list   train
6:       TRUE    <list[0]>         1      numeric   train

Since hyperparameter tuning is not the topic, let’s just set some basic things to demonstrate how this information is accessed and modified. The method for this is param_set$values:

# hyperparameters
l_xgboost$param_set$values = list(
  # making the algorithm train more slowly
  eta = 0.1,
  # limiting tree depth
  max_depth = 5,
  # maximum number of iterations
  nrounds = 100,
  # required with current xgboost versions
  verbose = 1
)

# checking
l_xgboost

── <LearnerRegrXgboost> (regr.xgboost): Extreme Gradient Boosting ──────────────
• Model: -
• Parameters: eta=0.1, max_depth=5, nrounds=100, verbose=1
• Validate: NULL
• Packages: mlr3, mlr3learners, and xgboost
• Predict Types: [response]
• Feature Types: logical, integer, and numeric
• Encapsulation: none (fallback: -)
• Properties: hotstart_forward, importance, internal_tuning, missings, offset,
validation, and weights
• Other settings: use_weights = 'use'

3 TRAINING AND PREDICTION

The next steps are training and prediction—we’ll cover tuning and resampling in future posts. First, split the dataset into training and test sets. For this, we’ll use the sample() function on two methods of the task_swiss object, row_ids and nrow. The first enumerates the indices of each row:

# row_ids method
task_swiss$row_ids

 [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
[26] 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47

While the second returns the number of rows in the dataset:

# nrow method
task_swiss$nrow

[1] 47

Thus, we can select the dataset indices into two random samples:

# ensuring reproducibility
set.seed(1)

# indices for training sample
train_set = sample(task_swiss$row_ids, 0.7 * task_swiss$nrow)

# indices for test sample
test_set = setdiff(task_swiss$row_ids, train_set)

# checking
head(train_set)

[1]  4 39  1 34 23 14

With the indices selected, we can train only on the randomly chosen 70% of the sample, without copying the data and allocating unnecessary memory:

# training
l_xgboost$train(task_swiss, row_ids = train_set)

# checking
l_xgboost$model

##### xgb.Booster
call:
  xgboost::xgb.train(params = pv[names(pv) %in% formalArgs(xgboost::xgb.params)], 
    data = xgb_data, nrounds = pv$nrounds, evals = pv$evals, 
    custom_metric = pv$custom_metric, verbose = pv$verbose, print_every_n = pv$print_every_n, 
    early_stopping_rounds = pv$early_stopping_rounds, maximize = pv$maximize, 
    save_period = pv$save_period, save_name = pv$save_name, callbacks = pv$callbacks %??% 
        list())
# of features: 5 
# of rounds:  100 

As we can see, in the first iteration the model obtained an rmse⁴. of 17.7, which is high considering the [1-100] scale of infant mortality. Throughout training, the error was reduced to 0.03, which does not mean that its performance will remain at this level when extrapolated to the test sample or new data, but it is a good sign. The expected result is that the real performance of the model, after being applied to the test sample, will be between the initial and final iteration. If it is better than the test performance, something is certainly wrong.

Let’s check the real performance after making predictions on the test sample. First, we pass the test indices to the learner object with the model and call the predict() method to get the predictions.

# predictions
preds = l_xgboost$predict(task_swiss, row_ids = test_set)

# checking
preds

── <PredictionRegr> for 15 observations: ───────────────────────────────────────
 row_ids truth response
       8  24.9 19.90614
      11  24.5 19.05639
      13  19.1 18.36767
     ---   ---      ---
      40  20.5 20.66445
      43  20.0 21.60975
      47  19.3 19.03525

In the predictions object, both the values predicted by the model (response) and the sample values (truth) are stored. These values can then be compared to calculate the model’s accuracy using the score() method:

# accuracy
preds$score(list(
  msr("regr.rmse"),
  msr("regr.mae")
))

regr.rmse  regr.mae 
 2.419703  1.889676 

# visualizing
autoplot(preds)

The model’s rmse on the test sample was only 2.59 units, which is very good performance!

4 INTERPRETATION

Since XGBoost has the feature importance property, we can extract it with the importance() method:

# feature importance
l_xgboost$importance()

  Fertility    Catholic Agriculture Examination   Education 
 0.43614832  0.37091549  0.09587702  0.07577311  0.02128606 

# visualizing
barplot(l_xgboost$importance())

However, importance alone does not describe the relationship of the feature with the response variable, nor its direction, being a very poor measure of interpretation. We will discuss interpretation techniques in other posts.

1 DATA

Here are two options to obtain this data:

1. Manually, by browsing ftp://ftp.mtps.gov.br/pdet/microdados/RAIS/2017/ES2017.7z
2. Programmatically via Python.

I chose Python here because I don’t know of a CRAN² package that handles 7zip compression—if you know one, please let me know in the comments! If you choose to download and extract via script, we can run Python code from R using {reticulate}:

# activating python
library(reticulate)

# selecting the environment
use_condaenv("base")

After activating the Python environment, let’s download and extract the dataset to a folder called data inside our project directory. If you downloaded it manually, create the data folder and place the file there.

# importing modules
import urllib.request as rq
import py7zr

# downloading data
url = "ftp://ftp.mtps.gov.br/pdet/microdados/RAIS/2017/ES2017.7z"
rq.urlretrieve(url, "data/rais.7z")

# extracting
rais = py7zr.SevenZipFile("data/rais.7z", mode="r")
rais.extractall(path="data/")
rais.close()

We will also need the Brazilian Occupation Code table to understand the professions of the people analyzed. After downloading it, place it in the data folder along with the Rais file.

With the data downloaded, let’s import, explore, and start making sense of it.

# packages
library(tidyverse)

# importing Rais
# as the data contains special characters (accents),
# it is necessary to set the encoding to LATIN1
data = read_delim("data/ES2017.txt", delim = ";",
                  locale = readr::locale(encoding = "LATIN1"),
                  col_names = FALSE,
                  skip = 1)

# selecting variables of interest and renaming columns
# we also need to replace commas with dots
# if you prefer, you can do this step via encoding
data = data %>%
  select(X8, X20, X26, X31, X35, X38) %>%
  rename("cod_profissao" = X8,
         "idade" = X20,
         "municipio" = X26,
         "cod_raca" = X31,
         "rem_media" = X35,
         "sexo" = X38) %>%
  mutate(rem_media = as.numeric(str_replace_all(rem_media, ",", ".")),
         sexo = as.integer(sexo),
         idade = as.integer(idade))

# importing CBO table
cbo = read_delim("data/CBO2002.csv", delim = ";",
                 locale = readr::locale(encoding = "LATIN1"),
                 col_names = c("cod_profissao", "profissao"),
                 col_types = c("d", "c"),
                 skip = 1)

# joining both tables and filtering only Vitória
data = data %>%
  inner_join(cbo, by = "cod_profissao") %>%
  filter(municipio == 320530)

Our data now looks like this:

# viewing dataframe
head(data)

# A tibble: 6 × 7
  cod_profissao idade municipio cod_raca rem_media  sexo profissao              
          <dbl> <int>     <dbl> <chr>        <dbl> <int> <chr>                  
1        717020    42    320530 04            909.     1 Servente de obras      
2        715210    28    320530 02           1167.     1 Pedreiro               
3        717020    63    320530 02           1022.     1 Servente de obras      
4        717020    59    320530 02            880.     1 Servente de obras      
5        517410    29    320530 08           1880.     1 Porteiro de edifícios  
6        715115    61    320530 08              0      1 Operador de escavadeira

2 AGGREGATE DIFFERENCES

First, let’s try to understand our sample:

# number of men and women
data %>%
  ggplot(aes(x = factor(sexo),
             fill = factor(sexo),
             label = scales::number(..count..))) +
  geom_bar() +
  geom_label(stat = "count",
             show.legend = FALSE,
             color = "grey30") +
  scale_y_continuous(labels = scales::number) +
  scale_fill_manual(name = "gender",
                    labels = c("men", "women"),
                    values = c("lightblue", "salmon")) +
  labs(x = "",
       y = "count",
       title = "NUMBER OF MEN AND WOMEN",
       subtitle = "sample from the city of Vitória-ES",
       caption = "source: Rais/2017") +
  theme_minimal() +
  theme(text = element_text(family = "Century Gothic",
                            color = "grey30"),
        axis.text.x = element_blank())

We have almost the same number of men and women in our sample. Before performing any further analysis or applying filters, let’s calculate the average income:

# average income
aggregate(data = data, rem_media ~ sexo, FUN = mean)

  sexo rem_media
1    1  3925.187
2    2  2846.727

In the aggregate, the average income of men in Vitória is almost 40% higher than that of women, and even when considering the median, it is still almost 30% higher. We can check if this difference is significant with a t-test, but first we should check if its assumptions are met.

# quartiles
boxplot_data = data %>%
  group_by(sexo) %>%
  summarise(stats = list(fivenum(rem_media))) %>%
  unnest(cols = c(stats))

# boxplot
data %>%
  ggplot(aes(
    x = factor(sexo),
    y = rem_media,
    fill = factor(sexo)
  )) +
  geom_boxplot() +
  geom_label(
    data = boxplot_data,
    aes(x = factor(sexo), y = stats, label = scales::number(stats)),
    nudge_x = 0.25,
    show.legend = FALSE
  ) +
  coord_cartesian(ylim = c(0, 5000)) +
  scale_y_continuous(labels = scales::number) +
  scale_fill_manual(
    name = "gender",
    labels = c("men", "women"),
    values = c("lightblue", "salmon")
  ) +
  labs(
    x = "",
    y = "average income",
    title = "DISTRIBUTION OF AVERAGE INCOME BETWEEN MEN AND WOMEN",
    subtitle = "sample from the city of Vitória-ES",
    caption = "source: Rais/2017"
  ) +
  theme_minimal() +
  theme(
    text = element_text(
      family = "Century Gothic",
      color = "grey30"
    ),
    axis.text.x = element_blank()
  )

# histogram
data %>%
  ggplot(aes(
    x = rem_media,
    fill = factor(sexo)
  )) +
  geom_histogram(binwidth = 500) +
  coord_cartesian(xlim = c(0, 20000)) +
  scale_y_continuous(labels = scales::number) +
  scale_x_continuous(labels = scales::number) +
  scale_fill_manual(
    name = "gender",
    labels = c("men", "women"),
    values = c("lightblue", "salmon")
  ) +
  labs(
    x = "average income",
    y = "count",
    title = "DISTRIBUTION OF AVERAGE INCOME BETWEEN MEN AND WOMEN",
    subtitle = "sample from the city of Vitória-ES",
    caption = "source: Rais/2017"
  ) +
  theme_minimal() +
  theme(text = element_text(
    family = "Century Gothic",
    color = "grey30"
  ))

# qqplot
# data will be on the line if normally distributed
par(mfrow = c(1, 2))

qqnorm(data[data$sexo == 1, ]$rem_media,
  main = "Q-Q PLOT: MEN")
qqline(data[data$sexo == 1, ]$rem_media)

qqnorm(data[data$sexo == 2, ]$rem_media,
  main = "Q-Q PLOT: WOMEN")
qqline(data[data$sexo == 2, ]$rem_media)

# ensure reproducibility
set.seed(1)

# number of samples
n = 1000

# means
means = rep(NA, n)

# drawing samples and calculating means
for (i in 1:n) {
  means[i] = mean(
    sample(data[data$sexo == 1, ]$rem_media,
      size = 30, replace = TRUE
    )
  )
}

# qqplot
qqnorm(means)
qqline(means)

# visualization
hist(means,
    main = "DISTRIBUTION OF SAMPLE MEANS",
    xlab = "means",
    sub = "seed = 1"
)

# transforming `sexo` variable
data = data %>%
  mutate(sexo = ifelse(sexo == 1, "man", "woman"))

# loading package
library(infer)

# calculating t statistic
calculated_stat = data %>%
  specify(rem_media ~ sexo) %>%
  calculate(stat = "t", order = c("man", "woman"))

# generating null distribution
null_dist = data %>%
  specify(rem_media ~ sexo) %>%
  hypothesise(null = "independence") %>%
  generate(reps = 100, type = "permute") %>%
  calculate(stat = "t", order = c("man", "woman"))

# visualizing null distribution and test statistic
null_dist %>%
  visualize(method = "both") +
  shade_p_value(calculated_stat, direction = "greater")

# calculating p-value
null_dist %>%
  get_p_value(obs_stat = calculated_stat, direction = "greater")

# A tibble: 1 × 1
  p_value
    <dbl>
1       0

# loading package
library(effectsize)

# calculating standardized difference
cohens_d(
  data[data$sexo == "man", ]$rem_media,
  data[data$sexo == "woman", ]$rem_media
)

Cohen's d |       95% CI
------------------------
0.26      | [0.25, 0.26]

- Estimated using pooled SD.

# interpreting d statistic
interpret_cohens_d(0.26, "gignac2016")

[1] "small"
(Rules: gignac2016)

rules(
  c(0.2, 0.41, 0.63),
  c("very small", "small", "moderate", "large"),
  "Gignac & Szodorai (2016)"
)

# Reference Thresholds (Gignac & Szodorai (2016))

         Label           
-------------------------
       very small <=  0.2
 0.2 <   small    <= 0.41
0.41 <  moderate  <= 0.63
0.63 <   large           

3 DIFFERENCES BETWEEN PROFESSIONS

We can extend our analysis by checking which professions have the greatest and smallest income gaps.

# creating dataframe of income differences
# selected professions
differences = data %>%
  mutate(profissao = case_when(
    str_detect(profissao, "Advogado") ~ "Lawyers",
    str_detect(profissao, "Engenheiro") ~ "Engineers",
    str_detect(profissao, "Médico") ~ "Doctors",
    str_detect(profissao, "Gerente") ~ "Managers",
    str_detect(profissao, "Dirigentes|Diretor") ~ "Executives",
    str_detect(profissao, "Economista") ~ "Economists",
    str_detect(profissao, "Analista") ~ "Analysts",
    str_detect(profissao, "Técnico") ~ "Technicians",
    str_detect(profissao, "Enfermeiro") ~ "Nurses",
    str_detect(profissao,
      "(Professor)(.*)(ensino superior)(.*)") ~ "Higher Ed. Professors",
    TRUE ~ "Others"
  )) %>%
  group_by(sexo, profissao) %>%
  summarise(
    renda = mean(rem_media),
    n = n()
  ) %>%
  ungroup() %>%
    pivot_wider(names_from = sexo, values_from = c(renda, n)) %>%
    mutate(diff = renda_man - renda_woman) %>%
    arrange(desc(diff))

# viewing table
differences

# A tibble: 11 × 6
   profissao             renda_man renda_woman  n_man n_woman  diff
   <chr>                     <dbl>       <dbl>  <int>   <int> <dbl>
 1 Executives               13559.       6994.    393     228 6565.
 2 Engineers                13997.      10446.   1850     409 3550.
 3 Lawyers                   8625.       5658.    244     342 2966.
 4 Economists                8251.       5576.    168     184 2675.
 5 Managers                  6599.       4463.   3659    3477 2136.
 6 Technicians               4790.       2731.  10434   10493 2060.
 7 Analysts                  5647.       4039.   2931    1869 1607.
 8 Higher Ed. Professors     7384.       6470.   1248    1062  914.
 9 Others                    3427.       2606. 121362  120682  820.
10 Doctors                   8232.       7704.   1840    1984  528.
11 Nurses                    4166.       4439.    540    2139 -273.

Among the selected professions, only nursing has a higher average income for women. Men have the advantage in all others. Let’s plot this:

# plot
mean_diff = 3925.18 - 2846.727
arrows = tibble(
  x1 = c(2.9, 7.6, 4.3),
  x2 = c(3.5, 7.2, 3.4),
  y1 = c(2000, 6000, 0),
  y2 = c(1200, 6300, -270)
)

differences %>%
  ggplot(aes(x = reorder(profissao, desc(profissao)), y = diff)) +
  geom_hline(yintercept = mean_diff, color = "grey85") +
  geom_segment(
    aes(y = mean_diff, yend = diff, xend = profissao),
    color = "grey85", size = .7
  ) +
  geom_point(aes(size = renda_man), color = "#F06337") +
  scale_y_continuous(labels = scales::number) +
  scale_size_continuous(labels = scales::number, range = c(1, 10)) +
  annotate("text",
    x = 2.5, y = 2000, size = 3.5, color = "grey85",
    label = "the average difference is\nR$ 1,078"
  ) +
  annotate ("text",
    x = 8.5, y = 5500, size = 3.5, color = "grey85",
    label = paste("executive positions",
    "have the largest difference",
    "and also the highest salaries",
    sep = "\n"
  )) +
  annotate("text",
  x = 5, y = 250, size = 3.5, color = "grey85",
  label = paste(
    "among these professions,",
    "only female nurses",
    "earn more than men",
    sep = "\n"),
  ) +
  geom_curve(
    data = arrows, aes(x = x1, y = y1, xend = x2, yend = y2),
    arrow = arrow(length = unit(0.1, "inch")),
    size = 0.3, color = "grey85", curvature = 0.3
  ) +
  coord_flip() +
  labs(
    x = "",
    y = "income difference in R$",
    size = "avg. income\n(men)",
    title = "gender inequality",
    subtitle = paste("salary disparities show the long road ahead to overcome the view",
    "of women as second-class professionals in the labor market",
    sep = "\n"),
    caption = "source: Rais 2017, Vitória/ES | elaboration: Alberson Miranda"
  ) +
  theme(
    panel.grid = element_blank(),
    plot.background = element_rect(fill = "#002538"),
    panel.background = element_rect(fill = "#002538"),
    text = element_text(
      family = "Century Gothic",
      color = "grey85"
    ),
    axis.text = element_text(color = "grey85", size = 10),
    axis.ticks = element_line(color = "grey85"),
    plot.title = element_text(size = 36, margin = margin(10, 0, 5, 10)),
    plot.subtitle = element_text(
      hjust = 0,
      margin = margin(0, 0, 30, 0),
      size = 16
    ),
    plot.caption = element_text(hjust = 1),
    plot.title.position = "plot",
    panel.border = element_rect(color = "grey85", fill = NA),
    legend.background = element_rect(fill = "#002538"),
    legend.key = element_rect(fill = "#002538")
  )

Among these selected professions, let’s analyze the effect size for doctors and executives (managers and directors).

# calculating t statistic
calculated_stat = data %>%
  filter(str_detect(profissao, "Médico")) %>%
  specify(rem_media ~ sexo) %>%
  calculate(stat = "t", order = c("man", "woman"))

# generating null distribution
null_dist = data %>%
  filter(str_detect(profissao, "Médico")) %>%
  specify(rem_media ~ sexo) %>%
  hypothesise(null = "independence") %>%
  generate(reps = 100, type = "permute") %>%
  calculate(stat = "t", order = c("man", "woman"))

# calculating confidence interval
percentile_ci = get_ci(null_dist)

# visualizing null distribution and test statistic
null_dist %>%
  visualize(method = "both") +
  shade_p_value(calculated_stat, direction = "greater") +
  shade_confidence_interval(endpoints = percentile_ci)

# calculating p-value
null_dist %>%
  get_p_value(obs_stat = calculated_stat, direction = "greater")

# A tibble: 1 × 1
  p_value
    <dbl>
1       0

 # calculating standardized difference
 cohens_d(
   data[data$sexo == "man" &
     str_detect(data$profissao, "Médico"), ]$rem_media,
   data[data$sexo == "woman" &
     str_detect(data$profissao, "Médico"), ]$rem_media
 )

Cohen's d |       95% CI
------------------------
0.12      | [0.06, 0.19]

- Estimated using pooled SD.

 # interpreting d statistic
 interpret_cohens_d(0.12, "gignac2016")

[1] "very small"
(Rules: gignac2016)

# calculating t statistic
calculated_stat = data %>%
  filter(str_detect(profissao, "Dirigentes|Diretor")) %>%
  specify(rem_media ~ sexo) %>%
  calculate(stat = "t", order = c("man", "woman"))

# generating null distribution
null_dist <- data %>%
  filter(str_detect(profissao, "Dirigentes|Diretor")) %>%
  specify(rem_media ~ sexo) %>%
  hypothesise(null = "independence") %>%
  generate(reps = 100, type = "permute") %>%
  calculate(stat = "t", order = c("man", "woman"))

# calculating confidence interval
percentile_ci <- get_ci(null_dist)

# visualizing null distribution and test statistic
null_dist %>%
  visualize(method = "both") +
  shade_p_value(calculated_stat, direction = "greater") +
  shade_confidence_interval(endpoints = percentile_ci)

 # calculating standardized difference
 cohens_d(
   data[data$sexo == "man" &
     str_detect(data$profissao, "Dirigentes|Diretor"), ]$rem_media,
   data[data$sexo == "woman" &
     str_detect(data$profissao, "Dirigentes|Diretor"), ]$rem_media
 )

Cohen's d |       95% CI
------------------------
0.47      | [0.30, 0.63]

- Estimated using pooled SD.

 # interpreting d statistic
 interpret_cohens_d(0.47, "gignac2016")

[1] "moderate"
(Rules: gignac2016)

4 FINAL REMARKS

The Gignac & Szodorai scale is just one rule of thumb among others—besides it, two widely used rules are Cohen’s (1988) and Sawilowsky’s (2009). It may be that for the nature of this problem, the interpretation suggested by these rules is not reasonable. In them, a d statistic of 0.47 is considered small or moderate. However, when it comes to gender pay disparities, we can argue that a rate of 69% of women earning less than the average man is a striking result, justifying a specific scale.

In any case, the inequality is clear and there are still several variables to be explored in this context—if we add race, what will the behavior be?

If you made it this far and have suggestions for variables, interpretation scales, or other professions to analyze, leave your comment and we’ll update with an edit!

1 THE TEST

The F test, among its various applications, is used together with the two-sample t-test—when it’s necessary to know whether the two sampled populations have the same variance or not.

It is also a parametric test, which means it assumes the populations are approximately normally distributed. Therefore, we must first ensure this assumption is met.

# density data
d1 = density(men)
d2 = density(women)

# splitting the grid into 2 columns
par(mfrow = c(1,2))

# visualization
plot(d1,
  main = "Density Plot: men")
polygon(d1, col = "lightblue")

plot(d2,
  main = "Density Plot: women")
polygon(d2, col = "salmon")

# logarithmic transformation
log_men = log(men)
log_women = log(women)

# calculating variance after transformation
var(log_men)

[1] 0.5614271

var(log_women)

[1] 0.5229548

# density data
d3 = density(log_men)
d4 = density(log_women)

# splitting the grid into 2 columns
par(mfrow = c(1,2))

# visualization
plot(d3,
  main = "Density Plot: log(men)")
polygon(d3, col = "lightblue")

plot(d4,
  main = "Density Plot: log(women)")
polygon(d4, col = "salmon")

# normality test
shapiro.test(log_men)

    Shapiro-Wilk normality test

data:  log_men
W = 0.98624, p-value = 0.5147

shapiro.test(log_women)

    Shapiro-Wilk normality test

data:  log_women
W = 0.98171, p-value = 0.2071

# degrees of freedom (n-1)
table(data[data$Housing == "rent",]$Sex)

female   male 
    95     84 

# F-statistic for the 95th percentile
qf(.95, 83, 94)

[1] 1.419123

# calculating critical value
var(log_men) / var(log_women)

[1] 1.073567

2 THE F TEST IN R

In base R, the syntax for the test is very similar to the t-test:

    F test to compare two variances

data:  log_men and log_women
F = 1.0736, num df = 83, denom df = 94, p-value = 0.7362
alternative hypothesis: true ratio of variances is not equal to 1
95 percent confidence interval:
 0.707196 1.638814
sample estimates:
ratio of variances 
          1.073567 

The test summary tells us that to reject the null hypothesis at = 5%, the ratio should be around 1.64 (variance of log_men greater than log_women) or 0.70 (variance of log_men less than log_women). Alternatively, we could reject the null hypothesis if we increased to 1-0.7362 = 26.38%, which is a probability of making a type II error too high to be considered reasonable.

1 THE PROBLEM

Consider a multiple-choice question with four possible answers. The question was designed to be very difficult, with none of the four answers considered wrong, but with only one correct answer. A test was conducted with 400 students. The test aims to verify whether more people answer the question correctly than would be expected just by chance (i.e., if everyone guessed the correct answer by pure luck).

Perform the hypothesis test (following the 5 steps), knowing that out of 400 students, 125 answered the question correctly. Use α = 2%.

I believe the biggest challenge here for beginner students is to understand the problem and structure it in a way that can be answered assertively. The key is in the following excerpt:

than would be expected just by chance (i.e., if everyone guessed the correct answer by pure luck).

The probability of choosing the correct answer among 4 options by chance is 25%. However, 125 out of 400 students got the question right, that is, 31.25%. The problem then is to infer whether the mean of 31.25% is significantly different from the expected mean, 25%, given the sample size and its variance. This can be achieved with the one-sample t-test.

2 A BIT OF SIMULATION

Before we get into the test itself, we know it is quite simple to calculate but its concept is not always so easily assimilated. To help, let’s simulate the data and try to visually represent what we are going to do.

Since these are binary data (correct or incorrect), we will use 1 for correct and 0 for incorrect.

# creating sample
data = c(rep(1, 125), rep(0, 275))

# to get the same random numbers as the example
set.seed(1)

# randomizing
data = sample(data)

# checking
data

  [1] 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 1 0 0 0 0 0 1 0 0 0 0 1 1 1 1 0 1 1 1 0 1
 [38] 0 0 1 0 0 0 1 0 1 1 0 0 0 1 0 0 0 0 1 0 1 0 0 0 0 1 1 1 0 0 0 1 0 1 0 0 0
 [75] 1 0 1 0 1 1 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 1 1 0 1 0 1 0 1 1
[112] 0 1 0 0 0 0 0 0 0 0 0 1 0 1 0 1 1 0 1 0 1 1 0 0 1 1 0 1 1 0 0 0 0 0 0 0 0
[149] 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 0 1 0
[186] 0 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1 0 0 0 1 1 0 0 1 1 1 0 0 1 0
[223] 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 1 0 0 1 0 1 0 0 1
[260] 0 0 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1 1 0 1 0 0 0 0 1
[297] 0 0 0 1 0 0 0 1 1 0 0 0 1 1 0 1 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1
[334] 1 1 0 0 1 1 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 1 1 0 1 1 0 1
[371] 0 0 0 1 0 0 0 0 0 1 0 0 0 0 1 1 0 0 1 0 0 1 0 0 1 0 0 1 1 0

Now that we have our vector with the random data of students who got the questions right and wrong, let’s visualize it:

# bar chart
par(mar = c(2, 2, 0, 0))
barplot(table(data), col = c("lightblue", "salmon"))

The intuition of the t-test is to say whether this difference between correct and incorrect answers occurs by mere chance in the sampling process or not. But what does that mean? In this case, the sample mean was 31.25% with 400 students. The characteristics of the distribution, the standard deviation, and the sample size allow us to attest whether the difference between the sample mean and the expected mean is significant?

To illustrate the reasoning, let’s take 10 samples of 10 students randomly and observe the behavior of the means. These samples of 10 students may have a mean of correct answers (red line) very close to the expected 25% (blue line) and far from the mean of the 400 students (green line).

# ensure reproducibility
set.seed(3)

# number of samples
n = 10

# means
means = rep(NA, n)

# taking samples and calculating means
for (i in 1:n) {
  means[i] = mean(
    sample(data, size = runif(1, min = 0, max = 10), replace = TRUE))
}

# visualization
hist(means,
     main = "1st draw: 10 samples of 10 students",
     xlab = "mean",
     sub = "seed = 3")
abline(v = mean(means, na.rm = TRUE), col = "red")
abline(v = 0.25, col = "blue")
abline(v = 0.3125, col = "green")
legend("topright", legend = c(
  expression(hat(mu)),
  expression(paste(mu, "= 0.25")),
  expression(paste(mu, "= 0.3325"))),
  col = c("red", "blue", "green"),
  lty = 1)

But it is also possible to take another draw of 10 samples with 10 different students and find a completely different mean, close to 45%:

# ensure reproducibility
set.seed(4)

# number of samples
n = 10

# means
means = rep(NA, n)

# taking samples and calculating means
for (i in 1:n) {
  means[i] = mean(
    sample(data, size = runif(1, min = 0, max = 10), replace = TRUE))
}

# visualization
hist(means,
     main = "2nd draw: 10 samples of 10 students",
     xlab = "mean",
     sub = "seed = 4")
abline(v = mean(means, na.rm = TRUE), col = "red")
abline(v = 0.25, col = "blue")
abline(v = 0.3125, col = "green")
legend("topright", legend = c(
  expression(hat(mu)),
  expression(paste(mu, "= 0.25")),
  expression(paste(mu, "= 0.3325"))),
  col = c("red", "blue", "green"),
  lty = 1)

As we increase the number of samples, the standard deviation stabilizes and the sample mean approaches the population mean. See the means of 50 samples of 10 students each:

# ensure reproducibility
set.seed(3)

# number of samples
n = 50

# means
means = rep(NA, n)

# taking samples and calculating means
for (i in 1:n) {
  means[i] = mean(
    sample(data, size = runif(1, min = 0, max = 10), replace = TRUE))
}

# separating the grid into 2 columns
par(mfrow = c(1, 2))

# visualization 1st draw
hist(means,
     main = "1st draw: 50 samples of 10 students",
     xlab = "mean",
     sub = "seed = 3")
abline(v = mean(means, na.rm = TRUE), col = "red")
abline(v = 0.25, col = "blue")
abline(v = 0.3125, col = "green")

################################################################################

# ensure reproducibility
set.seed(4)

# number of samples
n = 50

# means
means = rep(NA, n)

# taking samples and calculating means
for (i in 1:n) {
  means[i] = mean(
    sample(data, size = runif(1, min = 0, max = n), replace = TRUE))
}

# visualization 2nd draw
hist(means,
     main = "2nd draw: 50 samples of 10 students",
     xlab = "mean",
     sub = "seed = 4")
abline(v = mean(means, na.rm = TRUE), col = "red")
abline(v = 0.25, col = "blue")
abline(v = 0.3125, col = "green")
legend("topright", legend = c(
  expression(hat(mu)),
  expression(paste(mu, "= 0.25")),
  expression(paste(mu, "= 0.3325"))),
  col = c("red", "blue", "green"),
  lty = 1,
  cex = .7)

It can be observed that taking two draws of 50 samples of 10 students each, the means begin to converge to the mean observed in the total of 400 students.

And if we increase to 100 samples of 10 students?

# ensure reproducibility
set.seed(3)

# number of samples
n = 100

# means
means = rep(NA, n)

# taking samples and calculating means
for (i in 1:n) {
  means[i] = mean(
    sample(data, size = runif(1, min = 0, max = 10), replace = TRUE))
}

# separating the grid into 2 columns
par(mfrow = c(1, 2))

# visualization 1st draw
hist(means,
     main = "1st draw: 100 samples of 10 students",
     xlab = "mean",
     sub = "seed = 3")
abline(v = mean(means, na.rm = TRUE), col = "red")
abline(v = 0.25, col = "blue")
abline(v = 0.3125, col = "green")

################################################################################

# ensure reproducibility
set.seed(4)

# means
means = rep(NA, n)

# taking samples and calculating means
for (i in 1:n) {
  means[i] = mean(
    sample(data, size = runif(1, min = 0, max = 10), replace = TRUE))
}

# visualization 2nd draw
hist(means,
     main = "2nd draw: 100 samples of 10 students",
     xlab = "mean",
     sub = "seed = 4")
abline(v = mean(means, na.rm = TRUE), col = "red")
abline(v = 0.25, col = "blue")
abline(v = 0.3125, col = "green")
legend("topright", legend = c(
  expression(hat(mu)),
  expression(paste(mu, "= 0.25")),
  expression(paste(mu, "= 0.3325"))),
  col = c("red", "blue", "green"),
  lty = 1,
  cex = .7)

Notice that with 100 random samples, the means are already very close to the observed 33.25% in both draws. This happens because as the number of observations increases, the standard deviation tends to stabilize and the uncertainty decreases. It would be very difficult for the first 400¹ students to have a certain mean of correct answers and the next 50 students all get it right or wrong in a way that would generate changes in the standard deviation and significantly affect the mean. That is why, ultimately, the t-test is a sample size test. The question behind everything is: is my sample large enough for the difference to be significant?

Now, with the intuition of the test in mind, let’s formalize it.

3 THE TEST

To make the process very transparent and reinforce the concepts, it is always recommended to use the 5-step hypothesis testing framework, which are:

t.test(data, mu = 0.25, conf.level = 0.98)

    One Sample t-test

data:  data
t = 2.6934, df = 399, p-value = 0.00737
alternative hypothesis: true mean is not equal to 0.25
98 percent confidence interval:
 0.2583002 0.3666998
sample estimates:
mean of x 
   0.3125 

t.test(data, mu = 0.25, conf.level = 0.99263)

    One Sample t-test

data:  data
t = 2.6934, df = 399, p-value = 0.00737
alternative hypothesis: true mean is not equal to 0.25
99.263 percent confidence interval:
 0.2499995 0.3750005
sample estimates:
mean of x 
   0.3125 

t.test(data, mu = 0.2583, conf.level = 0.98)

    One Sample t-test

data:  data
t = 2.3357, df = 399, p-value = 0.02
alternative hypothesis: true mean is not equal to 0.2583
98 percent confidence interval:
 0.2583002 0.3666998
sample estimates:
mean of x 
   0.3125 

4 CONCLUSION

Now that we have all the tools, we can answer the student’s question from the forum. The teacher, yes, can be proud, because we can reject the hypothesis that the students’ mean correct answer rate was just luck. Congratulations to the class! :p

1 PROXY ADDRESS

The proxy server address is saved in a configuration file that can be obtained via your browser. If you are on Windows, you will have to access the infamous Internet Explorer, go to Internet Options, Connections, and LAN Settings.

Copy and paste the path shown in the Address field into your browser and you will download a .dat file with the proxy settings. Open the file in a reader¹. In the file, search for “PROXY”, this variable will contain the proxy server address in the format domain.com:port.

2 CHOOSE THE AUTHENTICATION METHOD

With the proxy address in hand, we need to choose how we will authenticate:

1. Enter the password for each push/pull
2. Save the password in .gitconfig

The first method applies when we are concerned about our password, that is, when the computer is shared or the configuration is saved on a network, for example.

When this is not the case, you can simply save the username and password in the configuration file and push/pull will occur directly.

# open the configuration file
usethis::edit_git_config()

[http]
    proxy = http[s]://domain.com:port

[credential]
    helper = wincred

[credential "helperselector"]
    selected = manager

[http]
    proxy = http[s]://user:password@domain.com:port